tribuchet: building on eliza Machine state will be reset. To keep it, pass --keep-machine-state start all VLans (finished: start all VLans, in 0.00 seconds) Test will time out and terminate in 900.0 seconds run the VM test script start all VMs client: starting vm lb: starting vm worker1: starting vm worker2: starting vm worker1 # Disk image does not exist, creating the virtualisation disk image... worker1 # Formatting '/build/vm-state-worker1/tmp.xFQxbUuM1N', fmt=raw size=1073741824 worker1 # mke2fs 1.47.4 (6-Mar-2025) lb # Disk image does not exist, creating the virtualisation disk image... worker1 # Discarding device blocks: 0/262144 done worker2 # Disk image does not exist, creating the virtualisation disk image... worker2: QEMU running (pid 52) client # Disk image does not exist, creating the virtualisation disk image... client: QEMU running (pid 49) lb: QEMU running (pid 50) lb # Formatting '/build/vm-state-lb/tmp.HSDNWdOjbH', fmt=raw size=1073741824 client # Formatting '/build/vm-state-client/tmp.19klYC5uS8', fmt=raw size=1073741824 worker2 # Formatting '/build/vm-state-worker2/tmp.de7i2iGXpJ', fmt=raw size=1073741824 lb # mke2fs 1.47.4 (6-Mar-2025) client # mke2fs 1.47.4 (6-Mar-2025) worker2 # mke2fs 1.47.4 (6-Mar-2025) lb # Discarding device blocks: 0/262144 done client # Discarding device blocks: 0/262144 done worker2 # Discarding device blocks: 0/262144 done lb # Creating filesystem with 262144 4k blocks and 65536 inodes client # Creating filesystem with 262144 4k blocks and 65536 inodes worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes lb # Filesystem UUID: 43eeae1b-fdb2-40d2-be15-b4a34cfa4c0d client # Filesystem UUID: b8f4d9fb-9828-4d03-a7aa-d8b67cf041ea worker2 # Filesystem UUID: fbdf700a-8a52-42f9-bc71-8eb921ea0479 lb # Superblock backups stored on blocks: client # Superblock backups stored on blocks: worker2 # Superblock backups stored on blocks: lb # 32768, 98304, 163840, 229376 client # 32768, 98304, 163840, 229376 worker2 # 32768, 98304, 163840, 229376 lb # client # worker2 # lb # Allocating group tables: 0/8 done client # Allocating group tables: 0/8 done worker2 # Allocating group tables: 0/8 done lb # Writing inode tables: 0/8 done client # Writing inode tables: 0/8 done worker2 # Writing inode tables: 0/8 done lb # Creating journal (8192 blocks): done client # Creating journal (8192 blocks): done worker2 # Creating journal (8192 blocks): done lb # Writing superblocks and filesystem accounting information: 0/8 done client # Writing superblocks and filesystem accounting information: 0/8 done worker2 # Writing superblocks and filesystem accounting information: 0/8 done lb # client # worker2 # lb # Virtualisation disk image created. client # Virtualisation disk image created. worker2 # Virtualisation disk image created. lb # Starting virtiofs daemons... client # Starting virtiofs daemons... worker2 # Starting virtiofs daemons... lb # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-22T16:53:39Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-22T16:53:39Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-22T16:53:39Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-22T16:53:39Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-22T16:53:39Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-22T16:53:39Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-22T16:53:39Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-22T16:53:39Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-22T16:53:39Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-22T16:53:39Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-22T16:53:39Z INFO virtiofsd] Client connected, servicing requests worker2 # [2026-09-22T16:53:39Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-22T16:53:39Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-22T16:53:39Z INFO virtiofsd] Client connected, servicing requests worker2 # [2026-09-22T16:53:39Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-22T16:53:39Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-22T16:53:39Z INFO virtiofsd] Client connected, servicing requests worker2 # [2026-09-22T16:53:39Z INFO virtiofsd] Client connected, servicing requests worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes worker1: QEMU running (pid 51) worker1 # Filesystem UUID: 4e060669-9984-4396-8f73-f143f0318f33 worker1 # Superblock backups stored on blocks: worker1 # 32768, 98304, 163840, 229376 worker1 # worker1 # Allocating group tables: 0/8 done (finished: start all VMs, in 0.51 seconds) worker1 # Writing inode tables: 0/8 done client: waiting for unit network-addresses-eth1.service worker1 # Creating journal (8192 blocks): done client: waiting for the VM to finish booting worker1 # Writing superblocks and filesystem accounting information: 0/8 done worker1 # worker1 # Virtualisation disk image created. worker1 # Starting virtiofs daemons... worker1 # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-22T16:53:39Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-22T16:53:39Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-22T16:53:39Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-22T16:53:39Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-22T16:53:39Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-22T16:53:39Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-22T16:53:39Z INFO virtiofsd] Client connected, servicing requests worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker1 # [ 0.000000] KASLR enabled worker1 # [ 0.000000] random: crng init done worker1 # [ 0.000000] Machine model: linux,dummy-virt worker1 # [ 0.000000] efi: UEFI not found. worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker1 # [ 0.000000] Zone ranges: worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] DMA32 empty worker1 # [ 0.000000] Normal empty worker1 # [ 0.000000] Device empty worker1 # [ 0.000000] Movable zone start for each node worker1 # [ 0.000000] Early memory node ranges worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker1 # [ 0.000000] psci: probing for conduit method from DT. worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker1 # [ 0.000000] psci: Trusted OS migration not required worker1 # [ 0.000000] psci: SMC Calling Convention v1.1 worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker1 # [ 0.000000] Detected PIPT I-cache on CPU0 worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface worker1 # [ 0.000000] CPU features: detected: Spectre-v4 worker1 # [ 0.000000] CPU features: detected: Spectre-BHB worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker1 # [ 0.000000] alternatives: applying boot alternatives lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/yms5bvxhm2a2zidkzddvi0xxaqsl6a35-nixos-system-worker1-test/init regInfo=/nix/store/vnq6mm2mwl8jxprixhfy00r13gsa7l45-closure-info/registration console=ttyAMA0,115200n8 console=tty0 lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 lb # [ 0.000000] KASLR enabled worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/vnq6mm2mwl8jxprixhfy00r13gsa7l45-closure-info/registration", will be passed to user space. lb # [ 0.000000] random: crng init done lb # [ 0.000000] Machine model: linux,dummy-virt worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes lb # [ 0.000000] efi: UEFI not found. lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB lb # [ 0.000000] Zone ranges: worker1 # [ 0.000000] software IO TLB: area num 1. lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB lb # [ 0.000000] DMA32 empty lb # [ 0.000000] Normal empty lb # [ 0.000000] Device empty worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) lb # [ 0.000000] Movable zone start for each node worker1 # [ 0.000000] Fallback order for Node 0: 0 lb # [ 0.000000] Early memory node ranges worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] Policy zone: DMA lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 lb # [ 0.000000] psci: probing for conduit method from DT. worker1 # [ 0.000000] allocated 3145728 bytes of page_ext lb # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups lb # [ 0.000000] psci: Trusted OS migration not required worker1 # [ 0.000000] rcu: Hierarchical RCU implementation. lb # [ 0.000000] psci: SMC Calling Convention v1.1 worker1 # [ 0.000000] rcu: RCU event tracing is enabled. lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled. lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker1 # [ 0.000000] Rude variant of Tasks RCU enabled. lb # [ 0.000000] Detected PIPT I-cache on CPU0 worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled. lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. lb # [ 0.000000] CPU features: detected: GICv3 CPU interface worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 lb # [ 0.000000] CPU features: detected: Spectre-v4 lb # [ 0.000000] CPU features: detected: Spectre-BHB worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] alternatives: applying boot alternatives worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker1 # [ 0.000000] GICv3: 256 SPIs implemented worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/qvl0xxkr124pdkzdvanphvcpy7197xra-nixos-system-lb-test/init regInfo=/nix/store/4y9byqv3ifqd7wzrdp7iwk6bnwmvvvvc-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/4y9byqv3ifqd7wzrdp7iwk6bnwmvvvvc-closure-info/registration", will be passed to user space. worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker2 # [ 0.000000] KASLR enabled lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker2 # [ 0.000000] random: crng init done worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker2 # [ 0.000000] Machine model: linux,dummy-virt lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker2 # [ 0.000000] efi: UEFI not found. lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] software IO TLB: area num 1. worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker2 # [ 0.000000] Zone ranges: worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker2 # [ 0.000000] DMA32 empty lb # [ 0.000000] Fallback order for Node 0: 0 worker2 # [ 0.000000] Normal empty worker2 # [ 0.000000] Device empty worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker2 # [ 0.000000] Movable zone start for each node lb # [ 0.000000] Policy zone: DMA worker2 # [ 0.000000] Early memory node ranges worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker1 # [ 0.000032] arm-pv: using stolen time PV worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] allocated 3145728 bytes of page_ext worker1 # [ 0.000650] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 client # [ 0.000000] KASLR enabled lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages worker1 # [ 0.000814] Console: colour dummy device 80x25 worker2 # [ 0.000000] psci: probing for conduit method from DT. client # [ 0.000000] random: crng init done lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker1 # [ 0.000822] printk: legacy console [tty0] enabled worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware. client # [ 0.000000] Machine model: linux,dummy-virt lb # [ 0.000000] rcu: Hierarchical RCU implementation. client # [ 0.000000] efi: UEFI not found. worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs lb # [ 0.000000] rcu: RCU event tracing is enabled. worker1 # [ 0.001015] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker2 # [ 0.000000] psci: Trusted OS migration not required client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker1 # [ 0.001021] pid_max: default: 32768 minimum: 301 lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker2 # [ 0.000000] psci: SMC Calling Convention v1.1 lb # [ 0.000000] Trampoline variant of Tasks RCU enabled. client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.001101] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker1 # [ 0.001258] landlock: Up and running. lb # [ 0.000000] Rude variant of Tasks RCU enabled. worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker1 # [ 0.001261] Yama: becoming mindful. client # [ 0.000000] Zone ranges: lb # [ 0.000000] Tracing variant of Tasks RCU enabled. worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker1 # [ 0.001907] LSM support for eBPF active worker2 # [ 0.000000] Detected PIPT I-cache on CPU0 client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. client # [ 0.000000] DMA32 empty worker1 # [ 0.002077] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.000000] Normal empty lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) client # [ 0.000000] Device empty worker1 # [ 0.002109] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.000000] Movable zone start for each node worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface worker1 # [ 0.003998] rcu: Hierarchical SRCU implementation. client # [ 0.000000] Early memory node ranges lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] CPU features: detected: Spectre-v4 worker1 # [ 0.004027] rcu: Max phase no-delay instances is 1000. worker2 # [ 0.000000] CPU features: detected: Spectre-BHB client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.005276] fsl-mc MSI: its@8080000 domain created worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker1 # [ 0.005367] EFI services will not be available. client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.005431] smp: Bringing up secondary CPUs ... client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker2 # [ 0.000000] alternatives: applying boot alternatives worker1 # [ 0.005440] smp: Brought up 1 node, 1 CPU client # [ 0.000000] psci: probing for conduit method from DT. lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker1 # [ 0.005443] SMP: Total of 1 processors activated. lb # [ 0.000000] GICv3: 256 SPIs implemented client # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker1 # [ 0.005448] CPU: All CPU(s) started at EL1 lb # [ 0.000000] GICv3: 0 Extended SPIs implemented client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs lb # [ 0.000000] Root IRQ handler: gic_handle_irq worker1 # [ 0.005459] CPU features: detected: Branch Target Identification client # [ 0.000000] psci: Trusted OS migration not required lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI client # [ 0.000000] psci: SMC Calling Convention v1.1 worker1 # [ 0.005464] CPU features: detected: ARMv8.4 Translation Table Level lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.005467] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/j24yd38mc0ckic4mmr56r3ka0bx8c52x-nixos-system-worker2-test/init regInfo=/nix/store/jw51lbh9icv07w9h8il9pf744i6nicyg-closure-info/registration console=ttyAMA0,115200n8 console=tty0 lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker1 # [ 0.005471] CPU features: detected: Data cache clean to the PoU not required for I/D coherence client # [ 0.000000] Detected PIPT I-cache on CPU0 lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker1 # [ 0.005474] CPU features: detected: Common not Private translations worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/jw51lbh9icv07w9h8il9pf744i6nicyg-closure-info/registration", will be passed to user space. client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker1 # [ 0.005477] CPU features: detected: CRC32 instructions worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) client # [ 0.000000] CPU features: detected: GICv3 CPU interface worker1 # [ 0.005480] CPU features: detected: Data cache clean to Point of Deep Persistence client # [ 0.000000] CPU features: detected: Spectre-v4 lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.005484] CPU features: detected: Data cache clean to Point of Persistence client # [ 0.000000] CPU features: detected: Spectre-BHB lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker1 # [ 0.005487] CPU features: detected: Data independent timing control (DIT) lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. worker1 # [ 0.005490] CPU features: detected: E0PD worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker2 # [ 0.000000] software IO TLB: area num 1. worker1 # [ 0.005492] CPU features: detected: Enhanced Counter Virtualization client # [ 0.000000] alternatives: applying boot alternatives lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker1 # [ 0.005495] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker1 # [ 0.005499] CPU features: detected: Enhanced Virtualization Traps worker2 # [ 0.000000] Fallback order for Node 0: 0 worker1 # [ 0.005502] CPU features: detected: Fine Grained Traps lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker2 # [ 0.000000] Policy zone: DMA worker1 # [ 0.005505] CPU features: detected: Generic authentication (architected QARMA5 algorithm) lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns lb # [ 0.000030] arm-pv: using stolen time PV worker1 # [ 0.005509] CPU features: detected: RCpc load-acquire (LDAPR) worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/c7dwh1z1cl7zl3zmm2gm23k4b20rxnfn-nixos-system-client-test/init regInfo=/nix/store/bvnhgdmv5qvfh2jgky9d95hczimkx039-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker1 # [ 0.005512] CPU features: detected: LSE atomic instructions worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 lb # [ 0.000511] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker1 # [ 0.005515] CPU features: detected: Privileged Access Never worker2 # [ 0.000000] allocated 3145728 bytes of page_ext lb # [ 0.000669] Console: colour dummy device 80x25 worker1 # [ 0.005518] CPU features: detected: PMUv3 client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/bvnhgdmv5qvfh2jgky9d95hczimkx039-closure-info/registration", will be passed to user space. worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages lb # [ 0.000676] printk: legacy console [tty0] enabled worker1 # [ 0.005521] CPU features: detected: RAS Extension Support worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker1 # [ 0.005524] CPU features: detected: RASv1p1 Extension Support worker2 # [ 0.000000] rcu: Hierarchical RCU implementation. lb # [ 0.000866] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker2 # [ 0.000000] rcu: RCU event tracing is enabled. client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.005526] CPU features: detected: Random Number Generator lb # [ 0.000873] pid_max: default: 32768 minimum: 301 worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker1 # [ 0.005529] CPU features: detected: Speculation barrier (SB) client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) lb # [ 0.000953] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled. worker1 # [ 0.005531] CPU features: detected: Stage-2 Force Write-Back lb # [ 0.001148] landlock: Up and running. client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker2 # [ 0.000000] Rude variant of Tasks RCU enabled. lb # [ 0.001151] Yama: becoming mindful. client # [ 0.000000] software IO TLB: area num 1. worker1 # [ 0.005534] CPU features: detected: TLB range maintenance instructions lb # [ 0.001688] LSM support for eBPF active worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled. client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker1 # [ 0.005538] CPU features: detected: Speculative Store Bypassing Safe (SSBS) lb # [ 0.001829] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker1 # [ 0.005578] alternatives: applying system-wide alternatives client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) lb # [ 0.001862] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 client # [ 0.000000] Fallback order for Node 0: 0 worker1 # [ 0.008543] CPU features: detected: BBM Level 2 without TLB conflict abort lb # [ 0.003796] rcu: Hierarchical SRCU implementation. worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 lb # [ 0.003802] rcu: Max phase no-delay instances is 1000. client # [ 0.000000] Policy zone: DMA lb # [ 0.005052] fsl-mc MSI: its@8080000 domain created worker1 # [ 0.008751] Memory: 1405672K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved) worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off lb # [ 0.005149] EFI services will not be available. worker1 # [ 0.009123] devtmpfs: initialized lb # [ 0.005240] smp: Bringing up secondary CPUs ... client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker1 # [ 0.010949] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.005247] smp: Brought up 1 node, 1 CPU client # [ 0.000000] allocated 3145728 bytes of page_ext worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 lb # [ 0.005250] SMP: Total of 1 processors activated. client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages worker1 # [ 0.010972] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker2 # [ 0.000000] GICv3: 256 SPIs implemented lb # [ 0.005256] CPU: All CPU(s) started at EL1 client # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented worker1 # [ 0.011142] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL lb # [ 0.005266] CPU features: detected: Branch Target Identification worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq worker1 # [ 0.011146] 0 pages in range for non-PLT usage client # [ 0.000000] rcu: Hierarchical RCU implementation. lb # [ 0.005270] CPU features: detected: ARMv8.4 Translation Table Level worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI client # [ 0.000000] rcu: RCU event tracing is enabled. worker1 # [ 0.011148] 508272 pages in range for PLT usage worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker1 # [ 0.011262] pinctrl core: initialized pinctrl subsystem client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. lb # [ 0.005273] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker1 # [ 0.012014] DMI not present or invalid. worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 client # [ 0.000000] Trampoline variant of Tasks RCU enabled. worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker1 # [ 0.015787] NET: Registered PF_NETLINK/PF_ROUTE protocol family client # [ 0.000000] Rude variant of Tasks RCU enabled. lb # [ 0.005277] CPU features: detected: Data cache clean to the PoU not required for I/D coherence client # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.017988] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations lb # [ 0.005281] CPU features: detected: Common not Private translations worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) lb # [ 0.005284] CPU features: detected: CRC32 instructions client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker1 # [ 0.018174] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) lb # [ 0.005287] CPU features: detected: Data cache clean to Point of Deep Persistence client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 worker1 # [ 0.018376] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 lb # [ 0.005290] CPU features: detected: Data cache clean to Point of Persistence worker1 # [ 0.018397] audit: initializing netlink subsys (disabled) client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 lb # [ 0.005293] CPU features: detected: Data independent timing control (DIT) worker1 # [ 0.018943] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. worker1 # [ 0.018945] thermal_sys: Registered thermal governor 'bang_bang' worker1 # [ 0.018949] thermal_sys: Registered thermal governor 'step_wise' client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker1 # [ 0.018951] thermal_sys: Registered thermal governor 'user_space' client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). client # [ 0.000000] GICv3: 256 SPIs implemented worker1 # [ 0.018957] thermal_sys: Registered thermal governor 'power_allocator' client # [ 0.000000] GICv3: 0 Extended SPIs implemented worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns client # [ 0.000000] Root IRQ handler: gic_handle_irq worker1 # [ 0.018989] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 worker1 # [ 0.018998] cpuidle: using governor ladder client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker1 # [ 0.019004] cpuidle: using governor menu client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker2 # [ 0.000029] arm-pv: using stolen time PV worker1 # [ 0.019205] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker1 # [ 0.019221] ASID allocator initialised with 65536 entries client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker2 # [ 0.000383] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker1 # [ 0.020453] Serial: AMBA PL011 UART driver worker2 # [ 0.000538] Console: colour dummy device 80x25 client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker2 # [ 0.000546] printk: legacy console [tty0] enabled worker1 # [ 0.025839] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 worker1 # [ 0.025998] printk: console [ttyAMA0] enabled worker2 # [ 0.000745] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) lb # [ 0.005296] CPU features: detected: E0PD worker2 # [ 0.000751] pid_max: default: 32768 minimum: 301 client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 lb # [ 0.005299] CPU features: detected: Enhanced Counter Virtualization worker2 # [ 0.000841] LSM: initializing lsm=capability,landlock,yama,bpf,ima client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker2 # [ 0.000974] landlock: Up and running. lb # [ 0.005302] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) worker2 # [ 0.000977] Yama: becoming mindful. client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. lb # [ 0.005306] CPU features: detected: Enhanced Virtualization Traps worker2 # [ 0.001404] LSM support for eBPF active lb # [ 0.005309] CPU features: detected: Fine Grained Traps worker2 # [ 0.001527] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns lb # [ 0.005312] CPU features: detected: Generic authentication (architected QARMA5 algorithm) client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker2 # [ 0.001559] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) lb # [ 0.005317] CPU features: detected: RCpc load-acquire (LDAPR) worker2 # [ 0.003348] rcu: Hierarchical SRCU implementation. lb # [ 0.005320] CPU features: detected: LSE atomic instructions worker2 # [ 0.003352] rcu: Max phase no-delay instances is 1000. client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns lb # [ 0.005323] CPU features: detected: Privileged Access Never worker2 # [ 0.004589] fsl-mc MSI: its@8080000 domain created lb # [ 0.005326] CPU features: detected: PMUv3 client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker2 # [ 0.004675] EFI services will not be available. client # [ 0.000026] arm-pv: using stolen time PV lb # [ 0.005328] CPU features: detected: RAS Extension Support worker2 # [ 0.004745] smp: Bringing up secondary CPUs ... worker2 # [ 0.004753] smp: Brought up 1 node, 1 CPU worker1 # [ 0.144401] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages lb # [ 0.005331] CPU features: detected: RASv1p1 Extension Support worker2 # [ 0.004756] SMP: Total of 1 processors activated. client # [ 0.000376] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) lb # [ 0.005334] CPU features: detected: Random Number Generator worker1 # [ 0.144417] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker2 # [ 0.004761] CPU: All CPU(s) started at EL1 client # [ 0.000524] Console: colour dummy device 80x25 lb # [ 0.005336] CPU features: detected: Speculation barrier (SB) worker1 # [ 0.144422] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages worker2 # [ 0.004771] CPU features: detected: Branch Target Identification client # [ 0.000531] printk: legacy console [tty0] enabled lb # [ 0.005339] CPU features: detected: Stage-2 Force Write-Back worker1 # [ 0.144426] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page worker2 # [ 0.004775] CPU features: detected: ARMv8.4 Translation Table Level lb # [ 0.005342] CPU features: detected: TLB range maintenance instructions client # [ 0.000715] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker1 # [ 0.144431] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker2 # [ 0.004779] CPU features: detected: Instruction cache invalidation not required for I/D coherence client # [ 0.000721] pid_max: default: 32768 minimum: 301 lb # [ 0.005347] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker1 # [ 0.144435] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page client # [ 0.000814] LSM: initializing lsm=capability,landlock,yama,bpf,ima lb # [ 0.005384] alternatives: applying system-wide alternatives worker2 # [ 0.004782] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker1 # [ 0.144439] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages client # [ 0.000930] landlock: Up and running. client # [ 0.000933] Yama: becoming mindful. lb # [ 0.008273] CPU features: detected: BBM Level 2 without TLB conflict abort worker2 # [ 0.004786] CPU features: detected: Common not Private translations worker1 # [ 0.144443] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page client # [ 0.001358] LSM support for eBPF active worker2 # [ 0.004789] CPU features: detected: CRC32 instructions client # [ 0.001504] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.004792] CPU features: detected: Data cache clean to Point of Deep Persistence lb # [ 0.008465] Memory: 1405636K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved) lb # [ 0.008812] devtmpfs: initialized client # [ 0.001536] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.004796] CPU features: detected: Data cache clean to Point of Persistence worker1 # [ 0.151607] fbcon: Taking over console client # [ 0.003374] rcu: Hierarchical SRCU implementation. lb # [ 0.010626] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker2 # [ 0.004798] CPU features: detected: Data independent timing control (DIT) worker1 # [ 0.151623] ACPI: Interpreter disabled. client # [ 0.003378] rcu: Max phase no-delay instances is 1000. worker2 # [ 0.004802] CPU features: detected: E0PD lb # [ 0.010647] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). client # [ 0.004590] fsl-mc MSI: its@8080000 domain created worker2 # [ 0.004804] CPU features: detected: Enhanced Counter Virtualization client # [ 0.004680] EFI services will not be available. lb # [ 0.010834] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL lb # [ 0.010838] 0 pages in range for non-PLT usage worker2 # [ 0.004808] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) client # [ 0.004743] smp: Bringing up secondary CPUs ... client # [ 0.004751] smp: Brought up 1 node, 1 CPU lb # [ 0.010839] 508272 pages in range for PLT usage worker2 # [ 0.004811] CPU features: detected: Enhanced Virtualization Traps client # [ 0.004754] SMP: Total of 1 processors activated. lb # [ 0.010948] pinctrl core: initialized pinctrl subsystem worker2 # [ 0.004814] CPU features: detected: Fine Grained Traps client # [ 0.004759] CPU: All CPU(s) started at EL1 lb # [ 0.011703] DMI not present or invalid. client # [ 0.004769] CPU features: detected: Branch Target Identification worker2 # [ 0.004818] CPU features: detected: Generic authentication (architected QARMA5 algorithm) lb # [ 0.015187] NET: Registered PF_NETLINK/PF_ROUTE protocol family client # [ 0.004772] CPU features: detected: ARMv8.4 Translation Table Level worker2 # [ 0.004823] CPU features: detected: RCpc load-acquire (LDAPR) lb # [ 0.017349] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker2 # [ 0.004826] CPU features: detected: LSE atomic instructions client # [ 0.004776] CPU features: detected: Instruction cache invalidation not required for I/D coherence lb # [ 0.017537] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker2 # [ 0.004828] CPU features: detected: Privileged Access Never worker2 # [ 0.004831] CPU features: detected: PMUv3 lb # [ 0.017733] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations client # [ 0.004779] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker2 # [ 0.004834] CPU features: detected: RAS Extension Support lb # [ 0.017754] audit: initializing netlink subsys (disabled) client # [ 0.004783] CPU features: detected: Common not Private translations worker2 # [ 0.004837] CPU features: detected: RASv1p1 Extension Support worker1 # [ 0.160390] iommu: Default domain type: Translated lb # [ 0.018333] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.004786] CPU features: detected: CRC32 instructions worker2 # [ 0.004839] CPU features: detected: Random Number Generator worker1 # [ 0.160399] iommu: DMA domain TLB invalidation policy: strict mode lb # [ 0.018335] thermal_sys: Registered thermal governor 'bang_bang' client # [ 0.004789] CPU features: detected: Data cache clean to Point of Deep Persistence worker2 # [ 0.004842] CPU features: detected: Speculation barrier (SB) worker1 # [ 0.160772] SCSI subsystem initialized lb # [ 0.018339] thermal_sys: Registered thermal governor 'step_wise' worker2 # [ 0.004845] CPU features: detected: Stage-2 Force Write-Back client # [ 0.004793] CPU features: detected: Data cache clean to Point of Persistence lb # [ 0.018341] thermal_sys: Registered thermal governor 'user_space' worker1 # [ 0.162684] usbcore: registered new interface driver usbfs worker2 # [ 0.004847] CPU features: detected: TLB range maintenance instructions client # [ 0.004796] CPU features: detected: Data independent timing control (DIT) lb # [ 0.018346] thermal_sys: Registered thermal governor 'power_allocator' worker1 # [ 0.162717] usbcore: registered new interface driver hub client # [ 0.004799] CPU features: detected: E0PD worker2 # [ 0.004852] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker1 # [ 0.162733] usbcore: registered new device driver usb lb # [ 0.018384] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 client # [ 0.004802] CPU features: detected: Enhanced Counter Virtualization worker2 # [ 0.004887] alternatives: applying system-wide alternatives worker1 # [ 0.163019] pps_core: LinuxPPS API ver. 1 registered lb # [ 0.018393] cpuidle: using governor ladder worker2 # [ 0.007751] CPU features: detected: BBM Level 2 without TLB conflict abort lb # [ 0.018399] cpuidle: using governor menu client # [ 0.004805] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) worker1 # [ 0.163025] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti client # [ 0.004808] CPU features: detected: Enhanced Virtualization Traps lb # [ 0.018602] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker1 # [ 0.163035] PTP clock support registered worker1 # [ 0.163084] EDAC MC: Ver: 3.0.0 client # [ 0.004812] CPU features: detected: Fine Grained Traps lb # [ 0.018617] ASID allocator initialised with 65536 entries worker2 # [ 0.007910] Memory: 1405652K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved) lb # [ 0.019852] Serial: AMBA PL011 UART driver worker2 # [ 0.008274] devtmpfs: initialized worker1 # [ 0.167484] scmi_core: SCMI protocol bus registered client # [ 0.004816] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker1 # [ 0.168412] FPGA manager framework worker2 # [ 0.010047] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) lb # [ 0.025223] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 client # [ 0.004820] CPU features: detected: RCpc load-acquire (LDAPR) lb # [ 0.025338] printk: console [ttyAMA0] enabled worker1 # [ 0.169323] vgaarb: loaded client # [ 0.004823] CPU features: detected: LSE atomic instructions worker2 # [ 0.010069] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). client # [ 0.004826] CPU features: detected: Privileged Access Never worker2 # [ 0.010278] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL worker1 # [ 0.169915] clocksource: Switched to clocksource arch_sys_counter client # [ 0.004829] CPU features: detected: PMUv3 lb # [ 0.145164] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages client # [ 0.004832] CPU features: detected: RAS Extension Support lb # [ 0.145180] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker2 # [ 0.010283] 0 pages in range for non-PLT usage client # [ 0.004835] CPU features: detected: RASv1p1 Extension Support worker2 # [ 0.010284] 508272 pages in range for PLT usage lb # [ 0.145186] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages client # [ 0.004838] CPU features: detected: Random Number Generator worker2 # [ 0.010396] pinctrl core: initialized pinctrl subsystem lb # [ 0.145190] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page worker2 # [ 0.011135] DMI not present or invalid. lb # [ 0.145194] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages client # [ 0.004840] CPU features: detected: Speculation barrier (SB) worker2 # [ 0.014240] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker1 # [ 0.173648] VFS: Disk quotas dquot_6.6.0 lb # [ 0.145199] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page client # [ 0.004843] CPU features: detected: Stage-2 Force Write-Back worker2 # [ 0.016359] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker1 # [ 0.173685] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) lb # [ 0.145203] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages client # [ 0.004846] CPU features: detected: TLB range maintenance instructions worker2 # [ 0.016528] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations lb # [ 0.145207] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page client # [ 0.004851] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker2 # [ 0.016719] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations client # [ 0.004887] alternatives: applying system-wide alternatives worker2 # [ 0.016739] audit: initializing netlink subsys (disabled) worker1 # [ 0.177265] netfs: FS-Cache loaded client # [ 0.007711] CPU features: detected: BBM Level 2 without TLB conflict abort worker2 # [ 0.017279] thermal_sys: Registered thermal governor 'fair_share' worker1 # [ 0.177394] pnp: PnP ACPI: disabled lb # [ 0.152433] fbcon: Taking over console worker2 # [ 0.017280] thermal_sys: Registered thermal governor 'bang_bang' lb # [ 0.152445] ACPI: Interpreter disabled. worker2 # [ 0.017284] thermal_sys: Registered thermal governor 'step_wise' client # [ 0.007901] Memory: 1405648K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved) client # [ 0.008251] devtmpfs: initialized worker2 # [ 0.017287] thermal_sys: Registered thermal governor 'user_space' client # [ 0.009986] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker2 # [ 0.017292] thermal_sys: Registered thermal governor 'power_allocator' worker2 # [ 0.017315] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 client # [ 0.010006] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker2 # [ 0.017325] cpuidle: using governor ladder worker1 # [ 0.181312] NET: Registered PF_INET protocol family client # [ 0.010187] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL worker2 # [ 0.017330] cpuidle: using governor menu client # [ 0.010191] 0 pages in range for non-PLT usage worker1 # [ 0.181586] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker2 # [ 0.017514] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. client # [ 0.010192] 508272 pages in range for PLT usage worker2 # [ 0.017528] ASID allocator initialised with 65536 entries client # [ 0.010278] pinctrl core: initialized pinctrl subsystem worker2 # [ 0.018699] Serial: AMBA PL011 UART driver client # [ 0.011013] DMI not present or invalid. lb # [ 0.159739] iommu: Default domain type: Translated client # [ 0.014005] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker2 # [ 0.023808] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 lb # [ 0.159748] iommu: DMA domain TLB invalidation policy: strict mode worker2 # [ 0.023926] printk: console [ttyAMA0] enabled client # [ 0.016147] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations lb # [ 0.161376] SCSI subsystem initialized client # [ 0.016322] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations lb # [ 0.162090] usbcore: registered new interface driver usbfs client # [ 0.016517] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations lb # [ 0.162122] usbcore: registered new interface driver hub client # [ 0.016537] audit: initializing netlink subsys (disabled) lb # [ 0.162138] usbcore: registered new device driver usb client # [ 0.017078] thermal_sys: Registered thermal governor 'fair_share' lb # [ 0.162417] pps_core: LinuxPPS API ver. 1 registered client # [ 0.017080] thermal_sys: Registered thermal governor 'bang_bang' lb # [ 0.162423] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti client # [ 0.017084] thermal_sys: Registered thermal governor 'step_wise' lb # [ 0.162433] PTP clock support registered client # [ 0.017086] thermal_sys: Registered thermal governor 'user_space' lb # [ 0.162480] EDAC MC: Ver: 3.0.0 client # [ 0.017092] thermal_sys: Registered thermal governor 'power_allocator' lb # [ 0.167043] scmi_core: SCMI protocol bus registered worker2 # [ 0.139306] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages client # [ 0.017132] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 worker2 # [ 0.139321] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page lb # [ 0.167990] FPGA manager framework client # [ 0.017143] cpuidle: using governor ladder lb # [ 0.168939] vgaarb: loaded worker2 # [ 0.139326] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages client # [ 0.017148] cpuidle: using governor menu lb # [ 0.169535] clocksource: Switched to clocksource arch_sys_counter worker2 # [ 0.139330] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page client # [ 0.017333] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. client # [ 0.017347] ASID allocator initialised with 65536 entries worker2 # [ 0.139334] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages client # [ 0.018540] Serial: AMBA PL011 UART driver worker2 # [ 0.139338] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page client # [ 0.023782] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 worker2 # [ 0.139343] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages client # [ 0.023933] printk: console [ttyAMA0] enabled worker2 # [ 0.139347] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page lb # [ 0.174419] VFS: Disk quotas dquot_6.6.0 worker2 # [ 0.146611] fbcon: Taking over console lb # [ 0.174457] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) worker2 # [ 0.146625] ACPI: Interpreter disabled. lb # [ 0.176087] netfs: FS-Cache loaded lb # [ 0.176211] pnp: PnP ACPI: disabled client # [ 0.141929] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages client # [ 0.141945] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page client # [ 0.141951] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages client # [ 0.141955] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page lb # [ 0.179991] NET: Registered PF_INET protocol family client # [ 0.141960] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages lb # [ 0.180245] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) client # [ 0.141964] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page worker2 # [ 0.155594] iommu: Default domain type: Translated client # [ 0.141969] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages worker2 # [ 0.155603] iommu: DMA domain TLB invalidation policy: strict mode client # [ 0.141973] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker2 # [ 0.155919] SCSI subsystem initialized worker2 # [ 0.157906] usbcore: registered new interface driver usbfs worker2 # [ 0.157938] usbcore: registered new interface driver hub client # [ 0.149498] fbcon: Taking over console client # [ 0.149513] ACPI: Interpreter disabled. worker2 # [ 0.157953] usbcore: registered new device driver usb worker2 # [ 0.158234] pps_core: LinuxPPS API ver. 1 registered worker2 # [ 0.158239] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker2 # [ 0.158250] PTP clock support registered worker2 # [ 0.158305] EDAC MC: Ver: 3.0.0 worker2 # [ 0.162861] scmi_core: SCMI protocol bus registered worker2 # [ 0.163812] FPGA manager framework worker2 # [ 0.164729] vgaarb: loaded worker2 # [ 0.165366] clocksource: Switched to clocksource arch_sys_counter worker1 # [ 0.210010] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) worker1 # [ 0.210046] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker1 # [ 0.210072] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) client # [ 0.158507] iommu: Default domain type: Translated worker1 # [ 0.210143] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) client # [ 0.158517] iommu: DMA domain TLB invalidation policy: strict mode client # [ 0.158873] SCSI subsystem initialized worker2 # [ 0.169351] VFS: Disk quotas dquot_6.6.0 worker1 # [ 0.210287] TCP: Hash tables configured (established 16384 bind 16384) worker1 # [ 0.210384] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) client # [ 0.160870] usbcore: registered new interface driver usbfs worker1 # [ 0.210415] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.160900] usbcore: registered new interface driver hub client # [ 0.160916] usbcore: registered new device driver usb worker1 # [ 0.210450] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) worker2 # [ 0.171864] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) client # [ 0.161191] pps_core: LinuxPPS API ver. 1 registered worker1 # [ 0.210566] NET: Registered PF_UNIX/PF_LOCAL protocol family worker2 # [ 0.172964] netfs: FS-Cache loaded worker1 # [ 0.210586] NET: Registered PF_XDP protocol family worker2 # [ 0.173074] pnp: PnP ACPI: disabled client # [ 0.161198] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker1 # [ 0.210601] PCI: CLS 0 bytes, default 64 client # [ 0.161208] PTP clock support registered worker1 # [ 0.210803] Trying to unpack rootfs image as initramfs... client # [ 0.161259] EDAC MC: Ver: 3.0.0 client # [ 0.165864] scmi_core: SCMI protocol bus registered client # [ 0.166851] FPGA manager framework client # [ 0.167792] vgaarb: loaded worker2 # [ 0.176821] NET: Registered PF_INET protocol family client # [ 0.168436] clocksource: Switched to clocksource arch_sys_counter worker2 # [ 0.177080] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker1 # [ 0.227460] kvm [1]: HYP mode not available client # [ 0.174630] VFS: Disk quotas dquot_6.6.0 client # [ 0.174674] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) client # [ 0.176296] netfs: FS-Cache loaded client # [ 0.176416] pnp: PnP ACPI: disabled lb # [ 0.208074] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) lb # [ 0.208113] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) lb # [ 0.208137] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) lb # [ 0.208206] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) client # [ 0.180235] NET: Registered PF_INET protocol family lb # [ 0.208348] TCP: Hash tables configured (established 16384 bind 16384) lb # [ 0.208424] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) lb # [ 0.208457] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.208492] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.208601] NET: Registered PF_UNIX/PF_LOCAL protocol family lb # [ 0.208621] NET: Registered PF_XDP protocol family client # [ 0.181132] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) lb # [ 0.208636] PCI: CLS 0 bytes, default 64 lb # [ 0.208855] Trying to unpack rootfs image as initramfs... lb # [ 0.223171] kvm [1]: HYP mode not available worker2 # [ 0.205137] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) worker2 # [ 0.205176] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker2 # [ 0.205202] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) worker2 # [ 0.205275] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker2 # [ 0.209239] TCP: Hash tables configured (established 16384 bind 16384) worker2 # [ 0.209316] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker2 # [ 0.209348] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) worker2 # [ 0.211815] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) worker2 # [ 0.211916] NET: Registered PF_UNIX/PF_LOCAL protocol family worker2 # [ 0.211935] NET: Registered PF_XDP protocol family worker2 # [ 0.211951] PCI: CLS 0 bytes, default 64 worker2 # [ 0.212163] Trying to unpack rootfs image as initramfs... worker2 # [ 0.220535] kvm [1]: HYP mode not available client # [ 0.209573] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) client # [ 0.209616] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) client # [ 0.209644] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) client # [ 0.209711] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) client # [ 0.209854] TCP: Hash tables configured (established 16384 bind 16384) client # [ 0.209933] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) client # [ 0.209965] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.210025] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.210122] NET: Registered PF_UNIX/PF_LOCAL protocol family client # [ 0.210147] NET: Registered PF_XDP protocol family client # [ 0.210163] PCI: CLS 0 bytes, default 64 client # [ 0.210397] Trying to unpack rootfs image as initramfs... client # [ 0.226315] kvm [1]: HYP mode not available worker1 # [ 0.318801] Initialise system trusted keyrings worker1 # [ 0.319549] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker1 # [ 0.320822] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker1 # [ 0.321584] 9p: Installing v9fs 9p2000 file system support lb # [ 0.314922] Initialise system trusted keyrings lb # [ 0.315650] workingset: timestamp_bits=42 max_order=19 bucket_order=0 lb # [ 0.316901] squashfs: version 4.0 (2009/01/31) Phillip Lougher lb # [ 0.317688] 9p: Installing v9fs 9p2000 file system support worker1 # [ 0.350262] Key type asymmetric registered worker1 # [ 0.350289] Asymmetric key parser 'x509' registered worker1 # [ 0.350354] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker1 # [ 0.352588] io scheduler mq-deadline registered worker1 # [ 0.352600] io scheduler kyber registered worker1 # [ 0.362045] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker2 # [ 0.312012] Initialise system trusted keyrings worker2 # [ 0.312737] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker1 # [ 0.363423] ledtrig-cpu: registered to indicate activity on CPUs worker2 # [ 0.314009] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker1 # [ 0.363842] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker1 # [ 0.363860] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker1 # [ 0.363885] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker2 # [ 0.314788] 9p: Installing v9fs 9p2000 file system support worker1 # [ 0.363894] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 lb # [ 0.345678] Key type asymmetric registered lb # [ 0.345700] Asymmetric key parser 'x509' registered worker1 # [ 0.363914] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits client # [ 0.309707] Initialise system trusted keyrings lb # [ 0.345764] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker1 # [ 0.363937] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] lb # [ 0.347853] io scheduler mq-deadline registered worker1 # [ 0.364011] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 lb # [ 0.347865] io scheduler kyber registered worker1 # [ 0.364021] pci_bus 0000:00: root bus resource [bus 00-ff] worker1 # [ 0.364027] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker1 # [ 0.364032] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.364037] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.364092] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.364521] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.316481] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker1 # [ 0.364702] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.364719] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.317827] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker1 # [ 0.364748] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.364763] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.365220] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.318612] 9p: Installing v9fs 9p2000 file system support worker1 # [ 0.365399] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.357690] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker1 # [ 0.365415] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.365443] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.365891] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint lb # [ 0.359152] ledtrig-cpu: registered to indicate activity on CPUs lb # [ 0.359533] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: lb # [ 0.359550] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 lb # [ 0.359562] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker1 # [ 0.386030] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] lb # [ 0.359571] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker1 # [ 0.386050] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.386080] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.338881] Key type asymmetric registered lb # [ 0.359591] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker2 # [ 0.338900] Asymmetric key parser 'x509' registered worker1 # [ 0.386549] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.359613] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker2 # [ 0.338958] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker1 # [ 0.386730] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.359703] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker2 # [ 0.340947] io scheduler mq-deadline registered worker1 # [ 0.386746] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.340958] io scheduler kyber registered lb # [ 0.359712] pci_bus 0000:00: root bus resource [bus 00-ff] worker1 # [ 0.386775] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.359718] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker1 # [ 0.386794] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.359723] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.387287] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint lb # [ 0.359728] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.387473] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.359783] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.387503] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.360221] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.387950] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint lb # [ 0.360405] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.388136] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.360422] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.388165] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.360452] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.339236] Key type asymmetric registered worker1 # [ 0.388562] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint lb # [ 0.360468] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.339253] Asymmetric key parser 'x509' registered worker1 # [ 0.388743] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] client # [ 0.339307] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) lb # [ 0.360923] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.388992] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.361105] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.389177] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.361120] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.389206] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.361149] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.389658] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.344484] io scheduler mq-deadline registered worker1 # [ 0.389843] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.344500] io scheduler kyber registered worker1 # [ 0.389872] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.354245] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker2 # [ 0.355510] ledtrig-cpu: registered to indicate activity on CPUs worker2 # [ 0.355863] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker2 # [ 0.355880] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker2 # [ 0.355892] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker2 # [ 0.355900] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 lb # [ 0.385650] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker2 # [ 0.355920] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits lb # [ 0.385844] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] lb # [ 0.385861] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.355941] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker1 # [ 0.414488] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.385890] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.356017] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker1 # [ 0.414692] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.386377] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.356026] pci_bus 0000:00: root bus resource [bus 00-ff] worker1 # [ 0.414723] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.386559] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.356032] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] lb # [ 0.386575] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.415218] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker2 # [ 0.356037] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.415510] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] lb # [ 0.386604] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.356041] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.415527] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.386622] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.356140] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.415557] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.387071] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker2 # [ 0.356563] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.356582] pl061_gpio 9030000.pl061: PL061 GPIO chip registered lb # [ 0.387257] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.416013] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker2 # [ 0.356742] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] client # [ 0.357224] ledtrig-cpu: registered to indicate activity on CPUs worker1 # [ 0.416195] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.387286] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.356758] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.416210] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.357604] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: lb # [ 0.387732] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker2 # [ 0.356787] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.416240] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.387916] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.357621] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker2 # [ 0.356803] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.416804] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned lb # [ 0.387946] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.357633] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker2 # [ 0.357236] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.416815] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned lb # [ 0.388339] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker2 # [ 0.357431] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] client # [ 0.357641] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker1 # [ 0.416821] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned lb # [ 0.388519] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker2 # [ 0.357446] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.357662] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker1 # [ 0.416866] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker2 # [ 0.357475] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.388769] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.416913] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker2 # [ 0.357920] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint client # [ 0.357684] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] lb # [ 0.388955] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.358099] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] lb # [ 0.388985] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.357754] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker1 # [ 0.416959] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker2 # [ 0.358115] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.357764] pci_bus 0000:00: root bus resource [bus 00-ff] lb # [ 0.389434] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.417006] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker2 # [ 0.358144] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.357770] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] lb # [ 0.389633] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.417053] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned client # [ 0.357775] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker2 # [ 0.358601] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.389662] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.358783] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.417099] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned client # [ 0.357780] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] lb # [ 0.390118] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.358798] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.417146] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned lb # [ 0.390306] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.357848] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker2 # [ 0.358827] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.417191] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned lb # [ 0.390335] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.358285] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.358845] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.358470] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.417237] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned lb # [ 0.390797] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker2 # [ 0.359289] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint client # [ 0.358486] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.391046] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker1 # [ 0.417304] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned worker2 # [ 0.359473] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.391063] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.358515] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.417350] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker2 # [ 0.359503] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.358531] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.391092] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.417371] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker2 # [ 0.359950] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint client # [ 0.358980] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.391549] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.417392] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker2 # [ 0.360134] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.359162] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.391731] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.417413] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned worker2 # [ 0.360164] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.359177] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.391747] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.417435] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker2 # [ 0.360543] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint client # [ 0.359206] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.391776] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.417456] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned worker2 # [ 0.360722] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] lb # [ 0.392338] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned client # [ 0.359650] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker1 # [ 0.417477] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned client # [ 0.359831] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] worker2 # [ 0.360992] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.392349] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker1 # [ 0.417503] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned worker2 # [ 0.361177] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.359846] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.392354] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker1 # [ 0.417524] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker2 # [ 0.361206] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.359875] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.417546] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned lb # [ 0.392397] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned client # [ 0.360318] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.417568] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned lb # [ 0.392443] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned client # [ 0.360522] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.417590] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned client # [ 0.360538] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.392488] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker1 # [ 0.417610] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned client # [ 0.360568] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.392533] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker1 # [ 0.417631] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned client # [ 0.360586] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.417652] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned lb # [ 0.392579] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker1 # [ 0.417673] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned client # [ 0.361041] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint lb # [ 0.392624] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker1 # [ 0.417694] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned client # [ 0.361244] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.392670] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker1 # [ 0.417715] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned client # [ 0.361273] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.417741] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.392715] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned client # [ 0.361730] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker1 # [ 0.417750] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] client # [ 0.361916] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.392761] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned worker1 # [ 0.417755] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] client # [ 0.361945] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.392829] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned worker1 # [ 0.418575] pci 0000:00:07.0: enabling device (0000 -> 0002) lb # [ 0.392874] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned client # [ 0.362359] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint client # [ 0.362538] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] lb # [ 0.392896] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker2 # [ 0.401748] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.392917] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned client # [ 0.362781] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.401947] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.392938] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned client # [ 0.362965] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.401978] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.392959] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned client # [ 0.362994] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.402445] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.392981] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned worker2 # [ 0.402631] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.363436] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.393002] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned client # [ 0.363620] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.402661] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.393026] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned client # [ 0.363649] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.403121] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint lb # [ 0.393047] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker2 # [ 0.403367] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] client # [ 0.364101] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.393069] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned worker2 # [ 0.403384] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.364285] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.393091] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker2 # [ 0.403413] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.364314] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.393113] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned worker2 # [ 0.403872] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.393135] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned worker2 # [ 0.404051] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.393156] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned worker2 # [ 0.404067] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.393177] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned worker2 # [ 0.404097] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.393198] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned worker2 # [ 0.404646] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned lb # [ 0.393219] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned worker2 # [ 0.404657] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned lb # [ 0.393240] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned lb # [ 0.393265] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] worker2 # [ 0.404663] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned lb # [ 0.393275] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] worker2 # [ 0.404705] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned lb # [ 0.393279] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] worker2 # [ 0.404751] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker2 # [ 0.404797] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker2 # [ 0.404844] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker2 # [ 0.404891] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker2 # [ 0.404937] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker2 # [ 0.404984] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker2 # [ 0.405029] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker2 # [ 0.405074] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned client # [ 0.412870] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker2 # [ 0.405141] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned client # [ 0.413146] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] lb # [ 0.454157] pci 0000:00:07.0: enabling device (0000 -> 0002) client # [ 0.413164] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.405185] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker1 # [ 0.480193] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker2 # [ 0.405207] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned client # [ 0.413194] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.405228] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned client # [ 0.413648] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker2 # [ 0.405248] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned client # [ 0.413828] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] client # [ 0.413843] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.405269] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker2 # [ 0.405290] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned client # [ 0.413873] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.405312] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned client # [ 0.414433] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker2 # [ 0.405336] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned client # [ 0.414445] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker2 # [ 0.405358] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned client # [ 0.414450] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned client # [ 0.414494] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned client # [ 0.414539] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned client # [ 0.414584] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned client # [ 0.414630] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned client # [ 0.414675] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned client # [ 0.414721] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker1 # [ 0.490123] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker1 # [ 0.493491] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) client # [ 0.414767] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned client # [ 0.414812] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker2 # [ 0.441427] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned client # [ 0.414857] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned worker2 # [ 0.441461] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned client # [ 0.414924] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned worker2 # [ 0.441484] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned client # [ 0.414970] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker2 # [ 0.441506] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned client # [ 0.414991] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker2 # [ 0.441527] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned client # [ 0.415012] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker2 # [ 0.441548] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned client # [ 0.415032] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned worker2 # [ 0.441569] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned client # [ 0.415053] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker2 # [ 0.441591] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned client # [ 0.415074] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned worker2 # [ 0.441613] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned worker2 # [ 0.441641] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] client # [ 0.415094] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned worker2 # [ 0.441650] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] client # [ 0.415119] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned worker2 # [ 0.441655] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] client # [ 0.415140] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker2 # [ 0.442474] pci 0000:00:07.0: enabling device (0000 -> 0002) client # [ 0.415161] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned lb # [ 0.480306] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker1 # [ 0.503242] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) client # [ 0.415183] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned client # [ 0.415205] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned client # [ 0.415225] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned client # [ 0.415245] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned client # [ 0.415266] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned worker1 # [ 0.507754] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) client # [ 0.415286] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned lb # [ 0.483375] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker1 # [ 0.509814] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) client # [ 0.415307] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned lb # [ 0.485295] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) client # [ 0.415327] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned client # [ 0.415352] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] client # [ 0.415361] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] client # [ 0.415365] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] client # [ 0.416155] pci 0000:00:07.0: enabling device (0000 -> 0002) lb # [ 0.494921] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) lb # [ 0.496882] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) worker1 # [ 0.523302] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker2 # [ 0.472230] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker1 # [ 0.525153] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker1 # [ 0.528477] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) lb # [ 0.508249] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) worker1 # [ 0.531148] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker1 # [ 0.533685] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker2 # [ 0.482588] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) client # [ 0.472464] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) client # [ 0.474499] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker2 # [ 0.486780] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) lb # [ 0.510765] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker2 # [ 0.488645] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) lb # [ 0.512476] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) lb # [ 0.514270] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) lb # [ 0.516096] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) client # [ 0.484533] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) client # [ 0.486892] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) worker1 # [ 0.551166] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker1 # [ 0.553654] msm_serial: driver initialized worker2 # [ 0.500268] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) worker1 # [ 0.553809] SuperH (H)SCI(F) driver initialized worker1 # [ 0.553863] STM32 USART driver initialized worker2 # [ 0.503006] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) lb # [ 0.530634] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker2 # [ 0.504652] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) client # [ 0.496989] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) client # [ 0.499377] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) client # [ 0.501112] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) client # [ 0.502899] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker2 # [ 0.514647] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker2 # [ 0.516537] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) lb # [ 0.546742] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled lb # [ 0.549257] msm_serial: driver initialized lb # [ 0.549397] SuperH (H)SCI(F) driver initialized lb # [ 0.549452] STM32 USART driver initialized worker2 # [ 0.527329] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker2 # [ 0.530454] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker1 # [ 0.587156] loop: module loaded worker1 # [ 0.587329] virtio_blk virtio2: 1/0/0 default/read/poll queues client # [ 0.520646] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) worker1 # [ 0.588111] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) client # [ 0.522521] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) client # [ 0.524695] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker2 # [ 0.543231] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker1 # [ 0.594482] megasas: 07.734.00.00-rc1 worker1 # [ 0.595269] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] client # [ 0.537653] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker1 # [ 0.597421] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.597457] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.545669] msm_serial: driver initialized client # [ 0.540052] msm_serial: driver initialized worker2 # [ 0.545813] SuperH (H)SCI(F) driver initialized client # [ 0.540202] SuperH (H)SCI(F) driver initialized worker2 # [ 0.545866] STM32 USART driver initialized client # [ 0.540255] STM32 USART driver initialized lb # [ 0.579093] loop: module loaded lb # [ 0.579254] virtio_blk virtio2: 1/0/0 default/read/poll queues lb # [ 0.579999] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker1 # [ 0.606854] Using buffer write method worker1 # [ 0.606918] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker1 # [ 0.609766] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.609787] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.586127] megasas: 07.734.00.00-rc1 lb # [ 0.586827] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] lb # [ 0.588919] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.588942] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.623051] Using buffer write method worker1 # [ 0.623105] Concatenating MTD devices: worker1 # [ 0.623109] (0): "0.flash" worker1 # [ 0.623113] (1): "0.flash" worker1 # [ 0.623119] into device "0.flash" lb # [ 0.598489] Using buffer write method lb # [ 0.598561] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] lb # [ 0.601368] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.601391] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.577592] loop: module loaded worker2 # [ 0.577755] virtio_blk virtio2: 1/0/0 default/read/poll queues worker2 # [ 0.578531] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) client # [ 0.571352] loop: module loaded client # [ 0.571520] virtio_blk virtio2: 1/0/0 default/read/poll queues client # [ 0.572337] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) lb # [ 0.614584] Using buffer write method lb # [ 0.614617] Concatenating MTD devices: lb # [ 0.614622] (0): "0.flash" lb # [ 0.614625] (1): "0.flash" lb # [ 0.614635] into device "0.flash" worker2 # [ 0.585926] megasas: 07.734.00.00-rc1 client # [ 0.576974] megasas: 07.734.00.00-rc1 worker2 # [ 0.586664] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] client # [ 0.577735] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] worker2 # [ 0.588816] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 client # [ 0.579840] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.588861] Intel/Sharp Extended Query Table at 0x0031 client # [ 0.579864] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.593878] Using buffer write method worker2 # [ 0.593945] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker2 # [ 0.595715] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.595737] Intel/Sharp Extended Query Table at 0x0031 client # [ 0.589403] Using buffer write method client # [ 0.589477] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] client # [ 0.592240] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 client # [ 0.592264] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.609427] Using buffer write method client # [ 0.601360] Using buffer write method worker2 # [ 0.609453] Concatenating MTD devices: worker2 # [ 0.609457] (0): "0.flash" client # [ 0.601386] Concatenating MTD devices: worker2 # [ 0.609461] (1): "0.flash" client # [ 0.601390] (0): "0.flash" client # [ 0.601394] (1): "0.flash" worker2 # [ 0.609469] into device "0.flash" client # [ 0.601404] into device "0.flash" worker1 # [ 0.848932] Freeing initrd memory: 26976K worker1 # [ 0.854984] tun: Universal TUN/TAP device driver, 1.6 lb # [ 0.833966] Freeing initrd memory: 26948K worker1 # [ 0.858666] thunder_xcv, ver 1.0 worker1 # [ 0.858711] thunder_bgx, ver 1.0 worker1 # [ 0.858734] nicpf, ver 1.0 worker1 # [ 0.859342] e1000: Intel(R) PRO/1000 Network Driver worker1 # [ 0.859350] e1000: Copyright (c) 1999-2006 Intel Corporation. worker1 # [ 0.859375] e1000e: Intel(R) PRO/1000 Network Driver worker1 # [ 0.859383] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. lb # [ 0.839983] tun: Universal TUN/TAP device driver, 1.6 worker1 # [ 0.859412] igb: Intel(R) Gigabit Ethernet Network Driver worker1 # [ 0.859418] igb: Copyright (c) 2007-2014 Intel Corporation. worker1 # [ 0.859441] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker1 # [ 0.859446] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker1 # [ 0.859584] sky2: driver version 1.30 worker1 # [ 0.861151] usbcore: registered new interface driver usb-storage lb # [ 0.843611] thunder_xcv, ver 1.0 lb # [ 0.843654] thunder_bgx, ver 1.0 worker1 # [ 0.861232] usbcore: registered new interface driver usbserial_generic lb # [ 0.843678] nicpf, ver 1.0 worker1 # [ 0.861246] usbserial: USB Serial support registered for generic lb # [ 0.844259] e1000: Intel(R) PRO/1000 Network Driver lb # [ 0.844266] e1000: Copyright (c) 1999-2006 Intel Corporation. worker1 # [ 0.861842] hv_vmbus: registering driver hyperv_keyboard lb # [ 0.844295] e1000e: Intel(R) PRO/1000 Network Driver worker1 # [ 0.862684] ehci-pci 0000:00:07.0: EHCI Host Controller lb # [ 0.844302] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker1 # [ 0.862710] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 lb # [ 0.844333] igb: Intel(R) Gigabit Ethernet Network Driver lb # [ 0.844339] igb: Copyright (c) 2007-2014 Intel Corporation. worker1 # [ 0.862913] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 lb # [ 0.844362] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker1 # [ 0.874546] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 lb # [ 0.844368] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker1 # [ 0.874850] hub 1-0:1.0: USB hub found lb # [ 0.844512] sky2: driver version 1.30 worker1 # [ 0.874866] hub 1-0:1.0: 6 ports detected lb # [ 0.853289] usbcore: registered new interface driver usb-storage worker1 # [ 0.878416] rtc-pl031 9010000.pl031: registered as rtc0 lb # [ 0.853447] usbcore: registered new interface driver usbserial_generic worker1 # [ 0.878444] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T16:53:41 UTC (1790096021) lb # [ 0.853461] usbserial: USB Serial support registered for generic worker1 # [ 0.878748] i2c_dev: i2c /dev entries driver lb # [ 0.854208] ehci-pci 0000:00:07.0: EHCI Host Controller lb # [ 0.854236] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 lb # [ 0.854491] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 lb # [ 0.859670] hv_vmbus: registering driver hyperv_keyboard worker1 # [ 0.884131] sdhci: Secure Digital Host Controller Interface driver lb # [ 0.861188] rtc-pl031 9010000.pl031: registered as rtc0 worker1 # [ 0.884142] sdhci: Copyright(c) Pierre Ossman worker1 # [ 0.884407] Synopsys Designware Multimedia Card Interface Driver lb # [ 0.861218] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T16:53:41 UTC (1790096021) worker1 # [ 0.884789] sdhci-pltfm: SDHCI platform and OF driver helper lb # [ 0.861520] i2c_dev: i2c /dev entries driver worker2 # [ 0.837086] Freeing initrd memory: 26944K worker1 # [ 0.888975] hid: raw HID events driver (C) Jiri Kosina worker1 # [ 0.889240] usbcore: registered new interface driver usbhid worker1 # [ 0.889247] usbhid: USB HID core driver worker1 # [ 0.892113] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available lb # [ 0.866919] sdhci: Secure Digital Host Controller Interface driver lb # [ 0.866929] sdhci: Copyright(c) Pierre Ossman worker1 # [ 0.893644] drop_monitor: Initializing network drop monitor service client # [ 0.832323] Freeing initrd memory: 26972K worker2 # [ 0.842975] tun: Universal TUN/TAP device driver, 1.6 lb # [ 0.867205] Synopsys Designware Multimedia Card Interface Driver worker1 # [ 0.893792] NET: Registered PF_INET6 protocol family lb # [ 0.867591] sdhci-pltfm: SDHCI platform and OF driver helper worker1 # [ 0.896781] Segment Routing with IPv6 lb # [ 0.869147] hid: raw HID events driver (C) Jiri Kosina worker1 # [ 0.896798] In-situ OAM (IOAM) with IPv6 lb # [ 0.869394] usbcore: registered new interface driver usbhid worker1 # [ 0.896826] NET: Registered PF_PACKET protocol family lb # [ 0.869400] usbhid: USB HID core driver lb # [ 0.874026] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 lb # [ 0.875015] hub 1-0:1.0: USB hub found client # [ 0.838231] tun: Universal TUN/TAP device driver, 1.6 lb # [ 0.875536] hub 1-0:1.0: 6 ports detected worker1 # [ 0.898663] 9pnet: Installing 9P2000 support worker1 # [ 0.898709] Key type dns_resolver registered worker2 # [ 0.846576] thunder_xcv, ver 1.0 lb # [ 0.877403] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker2 # [ 0.846619] thunder_bgx, ver 1.0 worker2 # [ 0.846642] nicpf, ver 1.0 worker2 # [ 0.847182] e1000: Intel(R) PRO/1000 Network Driver worker2 # [ 0.847189] e1000: Copyright (c) 1999-2006 Intel Corporation. lb # [ 0.880102] drop_monitor: Initializing network drop monitor service worker2 # [ 0.847217] e1000e: Intel(R) PRO/1000 Network Driver worker1 # [ 0.905314] registered taskstats version 1 lb # [ 0.880274] NET: Registered PF_INET6 protocol family worker2 # [ 0.847225] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker1 # [ 0.905465] Loading compiled-in X.509 certificates worker2 # [ 0.847255] igb: Intel(R) Gigabit Ethernet Network Driver client # [ 0.841855] thunder_xcv, ver 1.0 lb # [ 0.882177] Segment Routing with IPv6 client # [ 0.841897] thunder_bgx, ver 1.0 lb # [ 0.882194] In-situ OAM (IOAM) with IPv6 worker2 # [ 0.847260] igb: Copyright (c) 2007-2014 Intel Corporation. client # [ 0.841918] nicpf, ver 1.0 lb # [ 0.882221] NET: Registered PF_PACKET protocol family worker2 # [ 0.847284] igbvf: Intel(R) Gigabit Virtual Function Network Driver client # [ 0.842479] e1000: Intel(R) PRO/1000 Network Driver worker2 # [ 0.847290] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. client # [ 0.842486] e1000: Copyright (c) 1999-2006 Intel Corporation. worker2 # [ 0.847427] sky2: driver version 1.30 client # [ 0.842509] e1000e: Intel(R) PRO/1000 Network Driver worker2 # [ 0.848988] usbcore: registered new interface driver usb-storage client # [ 0.842519] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. lb # [ 0.883773] 9pnet: Installing 9P2000 support lb # [ 0.883822] Key type dns_resolver registered worker2 # [ 0.849068] usbcore: registered new interface driver usbserial_generic client # [ 0.842547] igb: Intel(R) Gigabit Ethernet Network Driver client # [ 0.842552] igb: Copyright (c) 2007-2014 Intel Corporation. worker2 # [ 0.849082] usbserial: USB Serial support registered for generic worker2 # [ 0.850066] ehci-pci 0000:00:07.0: EHCI Host Controller client # [ 0.842575] igbvf: Intel(R) Gigabit Virtual Function Network Driver client # [ 0.842580] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker2 # [ 0.850089] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 client # [ 0.842713] sky2: driver version 1.30 worker2 # [ 0.850355] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 client # [ 0.844271] usbcore: registered new interface driver usb-storage worker1 # [ 0.913984] Demotion targets for Node 0: null worker1 # [ 0.914081] Key type .fscrypt registered client # [ 0.844355] usbcore: registered new interface driver usbserial_generic worker2 # [ 0.861788] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker1 # [ 0.914087] Key type fscrypt-provisioning registered worker2 # [ 0.862084] hub 1-0:1.0: USB hub found client # [ 0.844370] usbserial: USB Serial support registered for generic worker1 # [ 0.914174] ima: No TPM chip found, activating TPM-bypass! worker2 # [ 0.862103] hub 1-0:1.0: 6 ports detected client # [ 0.845160] ehci-pci 0000:00:07.0: EHCI Host Controller worker1 # [ 0.914193] ima: Allocated hash algorithm: sha1 worker1 # [ 0.914214] ima: No architecture policies found client # [ 0.845191] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker2 # [ 0.864387] hv_vmbus: registering driver hyperv_keyboard client # [ 0.845380] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 lb # [ 0.890382] registered taskstats version 1 worker1 # [ 0.918156] input: gpio-keys as /devices/platform/gpio-keys/input/input0 lb # [ 0.890525] Loading compiled-in X.509 certificates client # [ 0.857055] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker2 # [ 0.865946] rtc-pl031 9010000.pl031: registered as rtc0 client # [ 0.857342] hub 1-0:1.0: USB hub found client # [ 0.857356] hub 1-0:1.0: 6 ports detected worker2 # [ 0.865969] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T16:53:41 UTC (1790096021) worker2 # [ 0.866276] i2c_dev: i2c /dev entries driver client # [ 0.859921] hv_vmbus: registering driver hyperv_keyboard client # [ 0.861405] rtc-pl031 9010000.pl031: registered as rtc0 lb # [ 0.899322] Demotion targets for Node 0: null lb # [ 0.899421] Key type .fscrypt registered client # [ 0.861430] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T16:53:41 UTC (1790096021) lb # [ 0.899427] Key type fscrypt-provisioning registered client # [ 0.861727] i2c_dev: i2c /dev entries driver worker2 # [ 0.871522] sdhci: Secure Digital Host Controller Interface driver lb # [ 0.899521] ima: No TPM chip found, activating TPM-bypass! worker2 # [ 0.871532] sdhci: Copyright(c) Pierre Ossman lb # [ 0.899540] ima: Allocated hash algorithm: sha1 worker2 # [ 0.871790] Synopsys Designware Multimedia Card Interface Driver lb # [ 0.899559] ima: No architecture policies found worker2 # [ 0.872152] sdhci-pltfm: SDHCI platform and OF driver helper lb # [ 0.903509] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker2 # [ 0.876275] hid: raw HID events driver (C) Jiri Kosina client # [ 0.866994] sdhci: Secure Digital Host Controller Interface driver worker2 # [ 0.876535] usbcore: registered new interface driver usbhid client # [ 0.867003] sdhci: Copyright(c) Pierre Ossman worker2 # [ 0.876541] usbhid: USB HID core driver client # [ 0.867271] Synopsys Designware Multimedia Card Interface Driver client # [ 0.867634] sdhci-pltfm: SDHCI platform and OF driver helper client # [ 0.871861] hid: raw HID events driver (C) Jiri Kosina worker2 # [ 0.879384] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available client # [ 0.872116] usbcore: registered new interface driver usbhid worker2 # [ 0.880883] drop_monitor: Initializing network drop monitor service client # [ 0.872123] usbhid: USB HID core driver worker2 # [ 0.881018] NET: Registered PF_INET6 protocol family worker2 # [ 0.884019] Segment Routing with IPv6 worker2 # [ 0.884040] In-situ OAM (IOAM) with IPv6 worker2 # [ 0.884067] NET: Registered PF_PACKET protocol family worker1 # [ 0.935904] clk: Disabling unused clocks worker1 # [ 0.935929] PM: genpd: Disabling unused power domains client # [ 0.875033] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available client # [ 0.877643] drop_monitor: Initializing network drop monitor service worker1 # [ 0.939998] Freeing unused kernel memory: 4736K client # [ 0.877814] NET: Registered PF_INET6 protocol family worker1 # [ 0.940188] Run /init as init process client # [ 0.879710] Segment Routing with IPv6 client # [ 0.879728] In-situ OAM (IOAM) with IPv6 worker2 # [ 0.885901] 9pnet: Installing 9P2000 support worker2 # [ 0.885984] Key type dns_resolver registered client # [ 0.879755] NET: Registered PF_PACKET protocol family worker2 # [ 0.892326] registered taskstats version 1 client # [ 0.881594] 9pnet: Installing 9P2000 support lb # [ 0.920507] clk: Disabling unused clocks worker2 # [ 0.892482] Loading compiled-in X.509 certificates client # [ 0.881637] Key type dns_resolver registered lb # [ 0.920530] PM: genpd: Disabling unused power domains lb # [ 0.924584] Freeing unused kernel memory: 4736K client # [ 0.888251] registered taskstats version 1 lb # [ 0.924783] Run /init as init process client # [ 0.888424] Loading compiled-in X.509 certificates worker2 # [ 0.900865] Demotion targets for Node 0: null worker2 # [ 0.900962] Key type .fscrypt registered worker2 # [ 0.900968] Key type fscrypt-provisioning registered worker2 # [ 0.901058] ima: No TPM chip found, activating TPM-bypass! worker2 # [ 0.901078] ima: Allocated hash algorithm: sha1 worker1 # [ 0.956076] systemd[1]: Successfully made /usr/ read-only. worker2 # [ 0.901097] ima: No architecture policies found worker2 # [ 0.905060] input: gpio-keys as /devices/platform/gpio-keys/input/input0 client # [ 0.896849] Demotion targets for Node 0: null client # [ 0.896944] Key type .fscrypt registered client # [ 0.896950] Key type fscrypt-provisioning registered client # [ 0.897037] ima: No TPM chip found, activating TPM-bypass! client # [ 0.897056] ima: Allocated hash algorithm: sha1 client # [ 0.897076] ima: No architecture policies found client # [ 0.901159] input: gpio-keys as /devices/platform/gpio-keys/input/input0 lb # [ 0.939645] systemd[1]: Successfully made /usr/ read-only. worker2 # [ 0.922014] clk: Disabling unused clocks worker2 # [ 0.922037] PM: genpd: Disabling unused power domains worker2 # [ 0.926099] Freeing unused kernel memory: 4736K worker2 # [ 0.926298] Run /init as init process client # [ 0.918054] clk: Disabling unused clocks client # [ 0.918073] PM: genpd: Disabling unused power domains client # [ 0.922146] Freeing unused kernel memory: 4736K client # [ 0.922352] Run /init as init process worker2 # [ 0.943455] systemd[1]: Successfully made /usr/ read-only. client # [ 0.938503] systemd[1]: Successfully made /usr/ read-only. worker1 # [ 1.121991] usb 1-1: new high-speed USB device number 2 using ehci-pci lb # [ 1.121613] usb 1-1: new high-speed USB device number 2 using ehci-pci worker2 # [ 1.109441] usb 1-1: new high-speed USB device number 2 using ehci-pci client # [ 1.104509] usb 1-1: new high-speed USB device number 2 using ehci-pci worker1 # [ 1.274536] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker1 # [ 1.290854] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 1.274067] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker1 # [ 1.302731] systemd[1]: Detected virtualization qemu. worker1 # [ 1.304722] systemd[1]: Detected architecture arm64. worker1 # [ 1.306677] systemd[1]: Running in initrd. worker1 # [ 1.309217] systemd[1]: Initializing machine ID from random generator. worker1 # [ 1.312146] systemd[1]: Hostname set to . lb # [ 1.280194] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 1.292648] systemd[1]: Detected virtualization qemu. lb # [ 1.294858] systemd[1]: Detected architecture arm64. worker2 # [ 1.263931] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 lb # [ 1.296760] systemd[1]: Running in initrd. client # [ 1.257011] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 lb # [ 1.299344] systemd[1]: Initializing machine ID from random generator. lb # [ 1.302150] systemd[1]: Hostname set to . worker2 # [ 1.278191] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 1.290089] systemd[1]: Detected virtualization qemu. worker2 # [ 1.292126] systemd[1]: Detected architecture arm64. client # [ 1.273239] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 1.294082] systemd[1]: Running in initrd. client # [ 1.285187] systemd[1]: Detected virtualization qemu. client # [ 1.287131] systemd[1]: Detected architecture arm64. worker2 # [ 1.296491] systemd[1]: Initializing machine ID from random generator. client # [ 1.289026] systemd[1]: Running in initrd. worker2 # [ 1.299287] systemd[1]: Hostname set to . client # [ 1.291526] systemd[1]: Initializing machine ID from random generator. client # [ 1.294362] systemd[1]: Hostname set to . worker1 # [ 1.358262] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 lb # [ 1.357787] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker2 # [ 1.345652] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 client # [ 1.340712] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker1 # [ 1.477971] usb 1-2: new high-speed USB device number 3 using ehci-pci lb # [ 1.477589] usb 1-2: new high-speed USB device number 3 using ehci-pci worker2 # [ 1.469414] usb 1-2: new high-speed USB device number 3 using ehci-pci client # [ 1.464486] usb 1-2: new high-speed USB device number 3 using ehci-pci worker1 # [ 1.612029] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 1.605500] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 1.638757] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker1 # [ 1.646063] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 client # [ 1.592210] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 1.606067] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 1.634358] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 lb # [ 1.639734] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 client # [ 1.621337] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.633990] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 client # [ 1.626949] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker2 # [ 1.639334] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker1 # [ 1.722029] systemd[1]: Queued start job for default target Initrd Default Target. worker1 # [ 1.732277] systemd[1]: Created slice Slice /system/modprobe. worker1 # [ 1.733462] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 1.734891] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker1 # [ 1.734928] systemd[1]: Reached target Path Units. worker1 # [ 1.734948] systemd[1]: Reached target Slice Units. worker1 # [ 1.734969] systemd[1]: Reached target Swaps. worker1 # [ 1.734987] systemd[1]: Reached target Timer Units. worker1 # [ 1.735178] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 1.735327] systemd[1]: Listening on Journal Socket (/dev/log). worker1 # [ 1.735467] systemd[1]: Listening on Journal Sockets. worker1 # [ 1.735595] systemd[1]: Listening on udev Control Socket. worker1 # [ 1.735706] systemd[1]: Listening on udev Kernel Socket. worker1 # [ 1.735728] systemd[1]: Reached target Socket Units. worker1 # [ 1.746685] systemd[1]: Starting Create List of Static Device Nodes... lb # [ 1.721870] systemd[1]: Queued start job for default target Initrd Default Target. worker1 # [ 1.747868] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 1.755260] systemd[1]: Mounting Kernel Configuration File System... lb # [ 1.732001] systemd[1]: Created slice Slice /system/modprobe. lb # [ 1.733226] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 1.734723] systemd[1]: Expecting device /dev/disk/by-label/nixos... lb # [ 1.734764] systemd[1]: Reached target Path Units. lb # [ 1.734785] systemd[1]: Reached target Slice Units. lb # [ 1.734804] systemd[1]: Reached target Swaps. lb # [ 1.734824] systemd[1]: Reached target Timer Units. lb # [ 1.735021] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 1.766220] systemd[1]: Starting Journal Service... lb # [ 1.735176] systemd[1]: Listening on Journal Socket (/dev/log). lb # [ 1.735321] systemd[1]: Listening on Journal Sockets. lb # [ 1.735462] systemd[1]: Listening on udev Control Socket. lb # [ 1.735574] systemd[1]: Listening on udev Kernel Socket. lb # [ 1.735596] systemd[1]: Reached target Socket Units. lb # [ 1.746924] systemd[1]: Starting Create List of Static Device Nodes... client # [ 1.710045] systemd[1]: Queued start job for default target Initrd Default Target. worker2 # [ 1.720313] systemd[1]: Queued start job for default target Initrd Default Target. lb # [ 1.748131] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 1.727598] systemd[1]: Created slice Slice /system/modprobe. client # [ 1.718829] systemd[1]: Created slice Slice /system/modprobe. worker2 # [ 1.728727] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 1.720058] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 1.757736] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 1.730016] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker2 # [ 1.730046] systemd[1]: Reached target Path Units. client # [ 1.721609] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker2 # [ 1.730064] systemd[1]: Reached target Slice Units. client # [ 1.721649] systemd[1]: Reached target Path Units. worker2 # [ 1.730082] systemd[1]: Reached target Swaps. client # [ 1.721671] systemd[1]: Reached target Slice Units. worker2 # [ 1.730100] systemd[1]: Reached target Timer Units. client # [ 1.721693] systemd[1]: Reached target Swaps. worker2 # [ 1.730298] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 1.721711] systemd[1]: Reached target Timer Units. worker2 # [ 1.730439] systemd[1]: Listening on Journal Socket (/dev/log). client # [ 1.721899] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 1.730569] systemd[1]: Listening on Journal Sockets. worker1 # [ 1.790356] systemd[1]: Starting Load Kernel Modules... lb # [ 1.765851] systemd[1]: Starting Journal Service... worker2 # [ 1.730692] systemd[1]: Listening on udev Control Socket. client # [ 1.722052] systemd[1]: Listening on Journal Socket (/dev/log). worker2 # [ 1.730795] systemd[1]: Listening on udev Kernel Socket. client # [ 1.722194] systemd[1]: Listening on Journal Sockets. worker1 # [ 1.791418] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 1.730814] systemd[1]: Reached target Socket Units. client # [ 1.722332] systemd[1]: Listening on udev Control Socket. client # [ 1.722446] systemd[1]: Listening on udev Kernel Socket. worker2 # [ 1.741077] systemd[1]: Starting Create List of Static Device Nodes... client # [ 1.722469] systemd[1]: Reached target Socket Units. worker2 # [ 1.741157] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 1.733878] systemd[1]: Starting Create List of Static Device Nodes... client # [ 1.735067] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 1.746744] systemd[1]: Mounting Kernel Configuration File System... client # [ 1.744632] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 1.810121] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 1.761799] systemd[1]: Starting Journal Service... client # [ 1.752733] systemd[1]: Starting Journal Service... lb # [ 1.789743] systemd[1]: Starting Load Kernel Modules... lb # [ 1.789854] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 1.819994] systemd-journald[72]: Collecting audit messages is disabled. worker1 # [ 1.826387] systemd[1]: Finished Create List of Static Device Nodes. worker1 # [ 1.827037] systemd[1]: Mounted Kernel Configuration File System. client # [ 1.776239] systemd[1]: Starting Load Kernel Modules... worker2 # [ 1.785542] systemd[1]: Starting Load Kernel Modules... worker2 # [ 1.785641] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 1.777366] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 1.817818] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 1.846114] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 1.826102] systemd-journald[72]: Collecting audit messages is disabled. worker2 # [ 1.797573] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 1.806942] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker2 # [ 1.808656] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev client # [ 1.792647] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 1.855109] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. lb # [ 1.837651] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 1.841916] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 1.842209] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 1.815942] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 1.816604] systemd[1]: Mounted Kernel Configuration File System. client # [ 1.810271] systemd-journald[72]: Collecting audit messages is disabled. worker1 # [ 1.878538] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 1.826832] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker1 # [ 1.880885] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 1.882840] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev client # [ 1.824577] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 1.834604] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker2 # [ 1.834831] [drm] features: -virgl +edid -resource_blob -host_visible worker2 # [ 1.834840] [drm] features: -context_init worker2 # [ 1.835525] [drm] number of scanouts: 1 worker2 # [ 1.835542] [drm] number of cap sets: 0 lb # [ 1.867224] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker1 # [ 1.895149] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker1 # [ 1.895385] [drm] features: -virgl +edid -resource_blob -host_visible worker1 # [ 1.895395] [drm] features: -context_init client # [ 1.832741] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker1 # [ 1.896076] [drm] number of scanouts: 1 worker1 # [ 1.896093] [drm] number of cap sets: 0 client # [ 1.832986] systemd[1]: Mounted Kernel Configuration File System. lb # [ 1.885638] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev client # [ 1.845759] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. lb # [ 1.888330] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 1.853665] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker2 # [ 1.853682] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 lb # [ 1.890717] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 1.910336] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker1 # [ 1.910352] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 worker2 # [ 1.870981] systemd-journald[72]: Collecting audit messages is disabled. client # [ 1.863980] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 1.866367] systemd[1]: Starting Create Static Device Nodes in /dev... lb # [ 1.902784] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 lb # [ 1.903009] [drm] features: -virgl +edid -resource_blob -host_visible worker2 # [ 1.872553] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 1.903018] [drm] features: -context_init lb # [ 1.903688] [drm] number of scanouts: 1 worker2 # [ 1.873980] systemd[1]: Starting Create Static Device Nodes in /dev... lb # [ 1.903705] [drm] number of cap sets: 0 worker1 # [ 1.934571] systemd[1]: Finished Create Static Device Nodes in /dev. worker1 # [ 1.934897] systemd[1]: Reached target Preparation for Local File Systems. worker1 # [ 1.934922] systemd[1]: Reached target Local File Systems. worker1 # [ 1.940894] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 1.876512] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev client # [ 1.882459] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 client # [ 1.882688] [drm] features: -virgl +edid -resource_blob -host_visible client # [ 1.882699] [drm] features: -context_init client # [ 1.883378] [drm] number of scanouts: 1 client # [ 1.883396] [drm] number of cap sets: 0 worker1 # [ 1.946377] Console: switching to colour frame buffer device 160x50 worker2 # [ 1.893956] Console: switching to colour frame buffer device 160x50 worker2 # [ 1.900520] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device lb # [ 1.925910] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic lb # [ 1.925927] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 client # [ 1.900759] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker1 # [ 1.970617] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device client # [ 1.900775] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 lb # [ 1.946192] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 1.917990] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 1.946378] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 1.918200] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 1.946404] systemd[1]: Reached target Local File Systems. worker2 # [ 1.918232] systemd[1]: Reached target Local File Systems. worker2 # [ 1.925728] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 1.953865] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker1 # [ 1.985629] systemd[1]: Started Journal Service. client # [ 1.925014] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 1.925204] systemd[1]: Reached target Preparation for Local File Systems. client # [ 1.925230] systemd[1]: Reached target Local File Systems. worker1 # [ 1.976384] systemd-modules-load[74]: Inserted module 'dm_mod' lb # [ 1.958829] Console: switching to colour frame buffer device 160x50 lb # [ 1.965372] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker1 # [ 1.978812] systemd-modules-load[74]: Module 'virtio_balloon' is built in client # [ 1.931343] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 1.933662] Console: switching to colour frame buffer device 160x50 worker1 # [ 1.991789] systemd-modules-load[74]: Module 'virtio_console' is built in worker1 # [ 1.993031] systemd-modules-load[74]: Inserted module 'virtio_gpu' worker1 # [ 1.994012] systemd-modules-load[74]: Module 'virtio_rng' is built in worker2 # [ 1.953588] systemd[1]: Finished Load Kernel Modules. lb # [ 1.982090] systemd[1]: Finished Load Kernel Modules. worker1 # [ 1.994963] systemd[1]: Finished Load Kernel Modules. worker1 # [ 1.995798] systemd[1]: Starting Apply Kernel Variables... lb # [ 1.985000] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 1.956727] systemd[1]: Starting Apply Kernel Variables... client # [ 1.961167] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker1 # [ 2.016152] systemd[1]: Starting Create System Files and Directories... lb # [ 1.996173] systemd-modules-load[74]: Inserted module 'dm_mod' lb # [ 1.997291] systemd-modules-load[74]: Module 'virtio_balloon' is built in lb # [ 1.998352] systemd-modules-load[74]: Module 'virtio_console' is built in lb # [ 1.999390] systemd-modules-load[74]: Inserted module 'virtio_gpu' client # [ 1.977657] systemd[1]: Finished Load Kernel Modules. lb # [ 2.013934] systemd[1]: Started Journal Service. client # [ 1.984855] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 2.033647] systemd-udevd[78]: Using default interface naming scheme 'v261'. lb # [ 2.016439] systemd-modules-load[74]: Module 'virtio_rng' is built in lb # [ 2.017507] systemd[1]: Starting Create System Files and Directories... worker2 # [ 2.009951] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 2.052373] systemd[1]: Finished Create System Files and Directories. worker1 # [ 2.054048] systemd[1]: Finished Apply Kernel Variables. lb # [ 2.031061] systemd[1]: Finished Apply Kernel Variables. client # [ 1.996396] systemd-modules-load[74]: Inserted module 'dm_mod' client # [ 2.013917] systemd[1]: Started Journal Service. client # [ 2.000432] systemd-modules-load[74]: Module 'virtio_balloon' is built in client # [ 2.001555] systemd-modules-load[74]: Module 'virtio_console' is built in client # [ 2.002596] systemd-modules-load[74]: Inserted module 'virtio_gpu' client # [ 2.003580] systemd-modules-load[74]: Module 'virtio_rng' is built in worker2 # [ 2.028750] systemd[1]: Started Journal Service. lb # [ 2.049443] systemd[1]: Finished Create System Files and Directories. worker1 # [ 2.076628] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 2.026357] systemd-modules-load[73]: Inserted module 'dm_mod' worker2 # [ 2.027586] systemd-modules-load[73]: Module 'virtio_balloon' is built in lb # [ 2.058156] systemd-udevd[78]: Using default interface naming scheme 'v261'. worker2 # [ 2.032672] systemd-modules-load[73]: Module 'virtio_console' is built in worker2 # [ 2.035687] systemd-modules-load[73]: Inserted module 'virtio_gpu' client # [ 2.028932] systemd[1]: Starting Create System Files and Directories... client # [ 2.030260] systemd-udevd[79]: Using default interface naming scheme 'v261'. worker2 # [ 2.044362] systemd-modules-load[73]: Module 'virtio_rng' is built in worker2 # [ 2.045646] systemd-udevd[78]: Using default interface naming scheme 'v261'. worker2 # [ 2.046709] systemd[1]: Starting Create System Files and Directories... client # [ 2.037335] systemd[1]: Finished Apply Kernel Variables. lb # [ 2.086917] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 2.049031] systemd[1]: Finished Create System Files and Directories. worker2 # [ 2.061832] systemd[1]: Finished Create System Files and Directories. worker1 # [ 2.123794] systemd[1]: Starting Virtual Console Setup... client # [ 2.064106] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 2.077155] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 2.144113] systemd[1]: Starting Virtual Console Setup... worker1 # [ 2.173227] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 2.123586] systemd[1]: Starting Virtual Console Setup... worker1 # [ 2.180193] systemd[1]: Finished Virtual Console Setup. client # [ 2.120231] systemd[1]: Starting Virtual Console Setup... lb # [ 2.196465] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 2.199716] systemd[1]: Finished Virtual Console Setup. worker2 # [ 2.177742] systemd-vconsole-setup[100]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 2.179702] systemd[1]: Finished Virtual Console Setup. client # [ 2.168449] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 2.171681] systemd[1]: Finished Virtual Console Setup. worker1 # [ 2.796703] systemd[1]: Finished Coldplug All udev Devices. worker1 # [ 2.798514] systemd[1]: Reached target System Initialization. worker1 # [ 2.804483] systemd[1]: Reached target Basic System. lb # [ 2.791263] systemd[1]: Finished Coldplug All udev Devices. lb # [ 2.792459] systemd[1]: Reached target System Initialization. lb # [ 2.793272] systemd[1]: Reached target Basic System. client # [ 2.761908] systemd[1]: Finished Coldplug All udev Devices. client # [ 2.762901] systemd[1]: Reached target System Initialization. client # [ 2.763730] systemd[1]: Reached target Basic System. worker2 # [ 2.782667] systemd[1]: Finished Coldplug All udev Devices. worker2 # [ 2.788367] systemd[1]: Reached target System Initialization. worker2 # [ 2.789246] systemd[1]: Reached target Basic System. worker1 # [ 2.935856] (udev-worker)[98]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 2.966016] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line. client # [ 2.912181] (udev-worker)[100]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 2.977931] (udev-worker)[98]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 2.946526] (udev-worker)[90]: Network interface NamePolicy= disabled on kernel command line. lb # [ 2.980377] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line. client # [ 2.939970] (udev-worker)[97]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 2.945872] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line. lb # [ 2.992976] (udev-worker)[97]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 2.970751] (udev-worker)[99]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. lb # [ 3.000949] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 2.975363] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.029865] systemd[1]: Found device /dev/disk/by-label/nixos. worker1 # [ 3.032155] systemd[1]: Reached target Initrd Root Device. worker1 # [ 3.035167] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... lb # [ 3.048808] systemd[1]: Found device /dev/disk/by-label/nixos. lb # [ 3.051982] systemd[1]: Reached target Initrd Root Device. lb # [ 3.055609] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... client # [ 3.024187] systemd[1]: Found device /dev/disk/by-label/nixos. worker2 # [ 3.036846] systemd[1]: Found device /dev/disk/by-label/nixos. client # [ 3.025155] systemd[1]: Reached target Initrd Root Device. worker2 # [ 3.041026] systemd[1]: Reached target Initrd Root Device. worker1 # [ 3.093271] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks client # [ 3.029270] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker2 # [ 3.045461] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker1 # [ 3.100155] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker1 # [ 3.103815] systemd[1]: Mounting /sysroot... lb # [ 3.109895] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks worker1 # [ 3.152821] EXT4-fs (vda): mounted filesystem 4e060669-9984-4396-8f73-f143f0318f33 r/w with ordered data mode. Quota mode: none. client # [ 3.077059] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks lb # [ 3.117551] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker1 # [ 3.146247] systemd[1]: Mounted /sysroot. client # [ 3.083663] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker1 # [ 3.149203] systemd[1]: Reached target Initrd Root File System. client # [ 3.088136] systemd[1]: Mounting /sysroot... worker1 # [ 3.152705] systemd[1]: Starting Mountpoints Configured in the Real Root... worker2 # [ 3.101572] systemd-fsck[109]: nixos: clean, 12/65536 files, 13019/262144 blocks lb # [ 3.131953] systemd[1]: Mounting /sysroot... worker2 # [ 3.107810] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker2 # [ 3.112277] systemd[1]: Mounting /sysroot... worker1 # [ 3.179785] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. worker1 # [ 3.186123] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... client # [ 3.138877] EXT4-fs (vda): mounted filesystem b8f4d9fb-9828-4d03-a7aa-d8b67cf041ea r/w with ordered data mode. Quota mode: none. worker1 # [ 3.191211] systemd[1]: Reloading... lb # [ 3.178870] EXT4-fs (vda): mounted filesystem 43eeae1b-fdb2-40d2-be15-b4a34cfa4c0d r/w with ordered data mode. Quota mode: none. client # [ 3.127598] systemd[1]: Mounted /sysroot. client # [ 3.130452] systemd[1]: Reached target Initrd Root File System. lb # [ 3.172193] systemd[1]: Mounted /sysroot. lb # [ 3.174039] systemd[1]: Reached target Initrd Root File System. client # [ 3.134737] systemd[1]: Starting Mountpoints Configured in the Real Root... worker2 # [ 3.158802] EXT4-fs (vda): mounted filesystem fbdf700a-8a52-42f9-bc71-8eb921ea0479 r/w with ordered data mode. Quota mode: none. lb # [ 3.176149] systemd[1]: Starting Mountpoints Configured in the Real Root... worker2 # [ 3.152544] systemd[1]: Mounted /sysroot. worker2 # [ 3.154159] systemd[1]: Reached target Initrd Root File System. worker2 # [ 3.157984] systemd[1]: Starting Mountpoints Configured in the Real Root... client # [ 3.159367] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. lb # [ 3.203613] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. client # [ 3.166082] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... client # [ 3.168672] systemd[1]: Reloading... worker2 # [ 3.183534] systemd-sysroot-fstab-check[117]: /sysroot should be mounted in the initrd, will request daemon-reload. lb # [ 3.212087] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... lb # [ 3.214801] systemd[1]: Reloading... worker2 # [ 3.188732] systemd[1]: Reload requested from client PID 117 ('systemd-sysroot') (unit initrd-parse-etc.service)... worker2 # [ 3.192730] systemd[1]: Reloading... worker1 # [ 3.395431] systemd[1]: Reloading finished in 206 ms. worker1 # [ 3.424923] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... worker1 # [ 3.428984] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... worker1 # [ 3.436186] systemd[1]: initrd-parse-etc.service: Deactivated successfully. client # [ 3.372145] systemd[1]: Reloading finished in 203 ms. worker1 # [ 3.437948] systemd[1]: Finished Mountpoints Configured in the Real Root. worker1 # [ 3.438894] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. lb # [ 3.417892] systemd[1]: Reloading finished in 206 ms. worker2 # [ 3.395836] systemd[1]: Reloading finished in 203 ms. client # [ 3.401433] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... client # [ 3.405927] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... lb # [ 3.451029] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... client # [ 3.411978] systemd[1]: initrd-parse-etc.service: Deactivated successfully. client # [ 3.415126] systemd[1]: Finished Mountpoints Configured in the Real Root. lb # [ 3.454999] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... client # [ 3.417461] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker2 # [ 3.430675] systemd-sysroot-fstab-check[117]: Requesting initrd-fs.target/start/replace... lb # [ 3.461246] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker2 # [ 3.434599] systemd-sysroot-fstab-check[117]: Requesting swap.target/start/replace... lb # [ 3.462657] systemd[1]: Finished Mountpoints Configured in the Real Root. lb # [ 3.463615] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker2 # [ 3.438277] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker2 # [ 3.440769] systemd[1]: Finished Mountpoints Configured in the Real Root. worker2 # [ 3.442762] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker1 # [ 3.812675] systemd[1]: Mounting /sysroot/nix/.ro-store... worker1 # [ 3.819561] systemd[1]: Mounting /sysroot/nix/.rw-store... lb # [ 3.807582] systemd[1]: Mounting /sysroot/nix/.ro-store... worker1 # [ 3.833445] systemd[1]: Mounting /sysroot/run... lb # [ 3.818562] systemd[1]: Mounting /sysroot/nix/.rw-store... worker1 # [ 3.848581] systemd[1]: Mounting /sysroot/tmp/shared... lb # [ 3.829573] systemd[1]: Mounting /sysroot/run... client # [ 3.793735] systemd[1]: Mounting /sysroot/nix/.ro-store... worker2 # [ 3.806644] systemd[1]: Mounting /sysroot/nix/.ro-store... client # [ 3.805658] systemd[1]: Mounting /sysroot/nix/.rw-store... lb # [ 3.849313] systemd[1]: Mounting /sysroot/tmp/shared... worker2 # [ 3.823576] systemd[1]: Mounting /sysroot/nix/.rw-store... worker2 # [ 3.827630] systemd[1]: Mounting /sysroot/run... client # [ 3.817625] systemd[1]: Mounting /sysroot/run... worker1 # [ 3.883000] systemd[1]: Mounting /sysroot/tmp/xchg... worker2 # [ 3.841395] systemd[1]: Mounting /sysroot/tmp/shared... lb # [ 3.871094] systemd[1]: Mounting /sysroot/tmp/xchg... client # [ 3.834900] systemd[1]: Mounting /sysroot/tmp/shared... worker2 # [ 3.862960] systemd[1]: Mounting /sysroot/tmp/xchg... client # [ 3.853878] systemd[1]: Mounting /sysroot/tmp/xchg... worker1 # [ 3.929192] systemd[1]: Mounted /sysroot/nix/.rw-store. worker1 # [ 3.935249] systemd[1]: Mounted /sysroot/run. lb # [ 3.911088] systemd[1]: Mounted /sysroot/nix/.rw-store. worker1 # [ 3.954471] fuse: init (API version 7.45) lb # [ 3.932906] fuse: init (API version 7.45) worker1 # [ 3.960905] virtiofs virtio6: discovered new tag: nix-store worker1 # [ 3.961699] virtiofs virtio6: virtio_fs_setup_dax: No cache capability worker1 # [ 3.958171] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 3.897042] systemd[1]: Mounted /sysroot/nix/.rw-store. lb # [ 3.949025] virtiofs virtio6: discovered new tag: nix-store worker1 # [ 3.975552] virtiofs virtio7: discovered new tag: shared worker1 # [ 3.976303] virtiofs virtio7: virtio_fs_setup_dax: No cache capability lb # [ 3.940828] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 3.956174] virtiofs virtio6: virtio_fs_setup_dax: No cache capability worker2 # [ 3.917744] systemd[1]: Mounted /sysroot/nix/.rw-store. worker1 # [ 3.985045] virtiofs virtio8: discovered new tag: xchg worker1 # [ 3.985795] virtiofs virtio8: virtio_fs_setup_dax: No cache capability worker2 # [ 3.933892] fuse: init (API version 7.45) client # [ 3.930371] fuse: init (API version 7.45) lb # [ 3.971190] virtiofs virtio7: discovered new tag: shared worker2 # [ 3.940445] virtiofs virtio6: discovered new tag: nix-store lb # [ 3.971922] virtiofs virtio7: virtio_fs_setup_dax: No cache capability worker2 # [ 3.941218] virtiofs virtio6: virtio_fs_setup_dax: No cache capability worker1 # [ 3.986644] systemd[1]: Mounted /sysroot/nix/.ro-store. client # [ 3.938915] virtiofs virtio6: discovered new tag: nix-store lb # [ 3.976555] virtiofs virtio8: discovered new tag: xchg client # [ 3.939706] virtiofs virtio6: virtio_fs_setup_dax: No cache capability lb # [ 3.977295] virtiofs virtio8: virtio_fs_setup_dax: No cache capability lb # [ 3.967832] systemd[1]: Mounted /sysroot/run. worker1 # [ 3.993763] systemd[1]: Mounted /sysroot/tmp/xchg. worker2 # [ 3.955065] virtiofs virtio7: discovered new tag: shared worker2 # [ 3.955816] virtiofs virtio7: virtio_fs_setup_dax: No cache capability client # [ 3.933236] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 3.998619] systemd[1]: Mounted /sysroot/tmp/shared. client # [ 3.954094] virtiofs virtio7: discovered new tag: shared worker2 # [ 3.963347] virtiofs virtio8: discovered new tag: xchg lb # [ 3.979585] systemd[1]: Mounted /sysroot/tmp/shared. client # [ 3.954845] virtiofs virtio7: virtio_fs_setup_dax: No cache capability worker2 # [ 3.964072] virtiofs virtio8: virtio_fs_setup_dax: No cache capability worker1 # [ 4.003744] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 3.941550] systemd[1]: Mounted /sysroot/run. worker1 # [ 4.006054] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 3.954051] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 3.983759] systemd[1]: Mounted /sysroot/tmp/xchg. client # [ 3.961688] virtiofs virtio8: discovered new tag: xchg client # [ 3.962404] virtiofs virtio8: virtio_fs_setup_dax: No cache capability lb # [ 3.987487] systemd[1]: Mounted /sysroot/nix/.ro-store. worker2 # [ 3.963792] systemd[1]: Mounted /sysroot/run. lb # [ 4.001317] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 4.002639] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 3.977471] systemd[1]: Mounted /sysroot/nix/.ro-store. worker2 # [ 3.979796] systemd[1]: Mounted /sysroot/tmp/shared. worker2 # [ 3.982050] systemd[1]: Mounted /sysroot/tmp/xchg. client # [ 3.973573] systemd[1]: Mounted /sysroot/nix/.ro-store. client # [ 3.976364] systemd[1]: Mounted /sysroot/tmp/shared. client # [ 3.978526] systemd[1]: Mounted /sysroot/tmp/xchg. client # [ 3.981620] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 3.984344] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.002163] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 4.003218] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.011082] systemd[1]: Mounting /sysroot/nix/store... worker2 # [ 4.043966] systemd[1]: Mounted /sysroot/nix/store. worker2 # [ 4.045838] systemd[1]: Reached target Initrd File Systems. worker2 # [ 4.051419] systemd[1]: Starting Find NixOS closure... worker2 # [ 4.058469] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker2 # [ 4.075987] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker2 # [ 4.093229] systemd[1]: Finished Find NixOS closure. worker2 # [ 4.095582] systemd[1]: Reached target Initrd Default Target. worker2 # [ 4.102433] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker2 # [ 4.126981] systemd[1]: Stopped target Initrd Default Target. worker2 # [ 4.130363] systemd[1]: Stopped target Basic System. worker2 # [ 4.131282] systemd[1]: Stopped target Initrd Root Device. worker2 # [ 4.134976] systemd[1]: Stopped target Path Units. worker2 # [ 4.136826] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker2 # [ 4.138887] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker2 # [ 4.141905] systemd[1]: Stopped target Slice Units. worker2 # [ 4.144003] systemd[1]: Stopped target Socket Units. worker2 # [ 4.149443] systemd[1]: Stopped target System Initialization. worker2 # [ 4.151274] systemd[1]: Stopped target Swaps. worker2 # [ 4.151950] systemd[1]: Stopped target Timer Units. worker2 # [ 4.154016] systemd[1]: dbus.socket: Deactivated successfully. worker2 # [ 4.160221] systemd[1]: Closed D-Bus System Message Bus Socket. worker2 # [ 4.162379] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker2 # [ 4.164373] systemd[1]: Stopped Find NixOS closure. worker2 # [ 4.165977] systemd[1]: Starting rw-sysroot-nix-store.service... worker2 # [ 4.169229] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker2 # [ 4.171967] systemd[1]: Stopped Apply Kernel Variables. worker2 # [ 4.174046] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker2 # [ 4.175800] systemd[1]: Stopped Load Kernel Modules. worker2 # [ 4.179966] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker2 # [ 4.184336] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker2 # [ 4.185608] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker2 # [ 4.189253] systemd[1]: Stopped Create System Files and Directories. worker2 # [ 4.193334] systemd[1]: Stopped target Local File Systems. worker2 # [ 4.195582] systemd[1]: Stopped target Preparation for Local File Systems. worker2 # [ 4.197569] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. worker2 # [ 4.198547] systemd[1]: Stopped Coldplug All udev Devices. worker2 # [ 4.199293] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker2 # [ 4.204237] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 4.206369] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 4.208229] systemd[1]: initrd-cleanup.service: Deactivated successfully. worker2 # [ 4.211372] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker2 # [ 4.212780] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 4.213776] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.214566] systemd[1]: systemd-udevd.service: Deactivated successfully. worker2 # [ 4.215474] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker2 # [ 4.216587] systemd[1]: systemd-udevd.service: Consumed 1.125s CPU time over 2.290s wall clock time, 20.8M memory peak. worker2 # [ 4.217986] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. worker2 # [ 4.218964] systemd[1]: Closed udev Control Socket. worker2 # [ 4.219657] systemd[1]: Starting Cleanup udev Database... worker2 # [ 4.223958] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker2 # [ 4.228371] systemd[1]: Stopped Create Static Device Nodes in /dev. worker2 # [ 4.229384] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker2 # [ 4.232189] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. worker2 # [ 4.233206] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker2 # [ 4.236128] systemd[1]: Stopped Create List of Static Device Nodes. worker2 # [ 4.257759] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker2 # [ 4.260478] systemd[1]: Finished Cleanup udev Database. worker2 # [ 4.261285] systemd[1]: Reached target Switch Root. worker2 # [ 4.264576] systemd[1]: Starting NixOS Activation... worker1 # [ 4.327263] (udev-worker)[95]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker1 # [ 4.335333] (udev-worker)[95]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory client # [ 4.283428] (udev-worker)[104]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker1 # [ 4.356771] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.291421] (udev-worker)[104]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory worker1 # [ 4.361576] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 4.364287] systemd[1]: Stopping Virtual Console Setup... worker1 # [ 4.365698] systemd[1]: Starting Virtual Console Setup... worker1 # [ 4.382027] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.383487] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 4.385618] systemd[1]: Starting Virtual Console Setup... worker2 # [ 4.338759] initrd-nixos-activation-start[170]: booting system configuration /nix/store/j24yd38mc0ckic4mmr56r3ka0bx8c52x-nixos-system-worker2-test client # [ 4.329486] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.331493] systemd[1]: Stopped Virtual Console Setup. client # [ 4.333809] systemd[1]: Stopping Virtual Console Setup... client # [ 4.334959] systemd[1]: Starting Virtual Console Setup... worker1 # [ 4.405312] systemd-vconsole-setup[155]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 4.408807] systemd[1]: Finished Virtual Console Setup. worker2 # [ 4.369776] initrd-nixos-activation-start[170]: running activation script... client # [ 4.360416] systemd-vconsole-setup[152]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 4.401122] (udev-worker)[92]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory client # [ 4.363798] systemd[1]: Finished Virtual Console Setup. lb # [ 4.407796] (udev-worker)[92]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory lb # [ 4.441226] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.442284] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.444138] systemd[1]: Stopping Virtual Console Setup... lb # [ 4.447863] systemd[1]: Starting Virtual Console Setup... lb # [ 4.462214] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.463644] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.468268] systemd[1]: Starting Virtual Console Setup... lb # [ 4.491847] systemd-vconsole-setup[156]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 4.495097] systemd[1]: Finished Virtual Console Setup. worker2 # [ 4.606920] initrd-nixos-activation-start[193]: setting up /etc... worker2 # [ 4.717735] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker2 # [ 4.720189] systemd[1]: Finished NixOS Activation. worker2 # [ 4.721603] systemd[1]: Starting Switch Root... worker2 # [ 4.742923] systemd[1]: Switching root. worker1 # [ 4.814019] systemd[1]: Mounting /sysroot/nix/store... lb # [ 4.810981] systemd[1]: Mounting /sysroot/nix/store... client # [ 4.795334] systemd[1]: Mounting /sysroot/nix/store... worker1 # [ 4.876849] systemd[1]: Mounted /sysroot/nix/store. worker1 # [ 4.880383] systemd[1]: Reached target Initrd File Systems. worker1 # [ 4.884610] systemd[1]: Starting Find NixOS closure... worker1 # [ 4.896566] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... lb # [ 4.875086] systemd[1]: Mounted /sysroot/nix/store. lb # [ 4.879257] systemd[1]: Reached target Initrd File Systems. lb # [ 4.883277] systemd[1]: Starting Find NixOS closure... lb # [ 4.892314] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... client # [ 4.857669] systemd[1]: Mounted /sysroot/nix/store. client # [ 4.860938] systemd[1]: Reached target Initrd File Systems. client # [ 4.865315] systemd[1]: Starting Find NixOS closure... client # [ 4.873070] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker1 # [ 4.937436] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker1 # [ 4.942799] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker1 # [ 4.956161] systemd[1]: Finished Find NixOS closure. worker1 # [ 4.958191] systemd[1]: Reached target Initrd Default Target. worker1 # [ 4.961764] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... lb # [ 4.937242] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker2 # [ 4.923114] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). lb # [ 4.943688] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. lb # [ 4.952960] systemd[1]: Finished Find NixOS closure. lb # [ 4.955701] systemd[1]: Reached target Initrd Default Target. lb # [ 4.957621] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... client # [ 4.917334] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. client # [ 4.922768] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker1 # [ 4.999119] systemd[1]: initrd-cleanup.service: Deactivated successfully. client # [ 4.936277] systemd[1]: Finished Find NixOS closure. worker1 # [ 5.001235] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. lb # [ 4.977505] systemd[1]: Stopped target Initrd Default Target. lb # [ 4.979647] systemd[1]: Stopped target Basic System. client # [ 4.939969] systemd[1]: Reached target Initrd Default Target. worker1 # [ 5.005378] systemd[1]: Stopped target Initrd Default Target. lb # [ 4.981090] systemd[1]: Stopped target Initrd Root Device. lb # [ 4.982310] systemd[1]: Stopped target Path Units. worker1 # [ 5.008200] systemd[1]: Stopped target Basic System. lb # [ 4.984266] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. client # [ 4.944577] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker1 # [ 5.010498] systemd[1]: Stopped target Initrd Root Device. worker1 # [ 5.012979] systemd[1]: Stopped target Path Units. lb # [ 4.988378] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. lb # [ 4.990089] systemd[1]: Stopped target Slice Units. lb # [ 4.992128] systemd[1]: Stopped target Socket Units. worker1 # [ 5.016229] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker1 # [ 5.020190] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. lb # [ 4.996176] systemd[1]: Stopped target System Initialization. lb # [ 4.997316] systemd[1]: Stopped target Swaps. lb # [ 4.998469] systemd[1]: Stopped target Timer Units. lb # [ 5.000484] systemd[1]: dbus.socket: Deactivated successfully. worker1 # [ 5.026167] systemd[1]: Stopped target Slice Units. worker1 # [ 5.027836] systemd[1]: Stopped target Socket Units. lb # [ 5.003957] systemd[1]: Closed D-Bus System Message Bus Socket. lb # [ 5.007417] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker1 # [ 5.032327] systemd[1]: Stopped target System Initialization. worker1 # [ 5.033287] systemd[1]: Stopped target Swaps. worker1 # [ 5.034016] systemd[1]: Stopped target Timer Units. worker1 # [ 5.034765] systemd[1]: dbus.socket: Deactivated successfully. worker1 # [ 5.035634] systemd[1]: Closed D-Bus System Message Bus Socket. lb # [ 5.011591] systemd[1]: Stopped Find NixOS closure. worker1 # [ 5.037740] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker1 # [ 5.038885] systemd[1]: Stopped Find NixOS closure. worker1 # [ 5.039642] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 4.976837] systemd[1]: Stopped target Initrd Default Target. client # [ 4.978572] systemd[1]: Stopped target Basic System. lb # [ 5.020918] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 5.045578] systemd[1]: systemd-sysctl.service: Deactivated successfully. lb # [ 5.021856] systemd[1]: systemd-sysctl.service: Deactivated successfully. client # [ 4.982471] systemd[1]: Stopped target Initrd Root Device. lb # [ 5.022926] systemd[1]: Stopped Apply Kernel Variables. client # [ 4.983704] systemd[1]: Stopped target Path Units. lb # [ 5.023684] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker1 # [ 5.050350] systemd[1]: Stopped Apply Kernel Variables. client # [ 4.985453] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. lb # [ 5.032180] systemd[1]: Stopped Load Kernel Modules. worker1 # [ 5.056384] systemd[1]: systemd-modules-load.service: Deactivated successfully. client # [ 4.992179] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. client # [ 4.993847] systemd[1]: Stopped target Slice Units. worker1 # [ 5.058477] systemd[1]: Stopped Load Kernel Modules. client # [ 4.994869] systemd[1]: Stopped target Socket Units. worker1 # [ 5.059546] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. lb # [ 5.035855] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. client # [ 4.996709] systemd[1]: Stopped target System Initialization. worker1 # [ 5.061065] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker1 # [ 5.062807] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker1 # [ 5.063865] systemd[1]: Stopped Create System Files and Directories. lb # [ 5.039256] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. client # [ 5.001504] systemd[1]: Stopped target Swaps. client # [ 5.003274] systemd[1]: Stopped target Timer Units. lb # [ 5.043143] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. lb # [ 5.044250] systemd[1]: Stopped Create System Files and Directories. worker1 # [ 5.069341] systemd[1]: Stopped target Local File Systems. worker1 # [ 5.072252] systemd[1]: Stopped target Preparation for Local File Systems. lb # [ 5.048172] systemd[1]: Stopped target Local File Systems. lb # [ 5.049029] systemd[1]: Stopped target Preparation for Local File Systems. client # [ 5.009579] systemd[1]: dbus.socket: Deactivated successfully. lb # [ 5.049966] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. client # [ 5.010670] systemd[1]: Closed D-Bus System Message Bus Socket. worker1 # [ 5.076178] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. client # [ 5.011675] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. lb # [ 5.051987] systemd[1]: Stopped Coldplug All udev Devices. worker1 # [ 5.077282] systemd[1]: Stopped Coldplug All udev Devices. lb # [ 5.053225] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker1 # [ 5.078077] systemd[1]: Stopping Rule-based Manager for Device Events and Files... lb # [ 5.054256] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 5.055254] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 5.080234] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 5.055979] systemd[1]: initrd-cleanup.service: Deactivated successfully. lb # [ 5.057031] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. lb # [ 5.057924] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 5.058910] systemd[1]: Finished rw-sysroot-nix-store.service. client # [ 5.019467] systemd[1]: Stopped Find NixOS closure. worker1 # [ 5.084156] systemd[1]: Stopped Virtual Console Setup. lb # [ 5.059742] systemd[1]: systemd-udevd.service: Deactivated successfully. client # [ 5.020433] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 5.084929] systemd[1]: systemd-udevd.service: Deactivated successfully. worker1 # [ 5.085835] systemd[1]: Stopped Rule-based Manager for Device Events and Files. client # [ 5.021599] systemd[1]: systemd-sysctl.service: Deactivated successfully. client # [ 5.022625] systemd[1]: Stopped Apply Kernel Variables. client # [ 5.023434] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker1 # [ 5.088200] systemd[1]: systemd-udevd.service: Consumed 1.387s CPU time over 3.121s wall clock time, 22.3M memory peak. worker1 # [ 5.092330] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. lb # [ 5.068205] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker1 # [ 5.093365] systemd[1]: Closed udev Control Socket. worker1 # [ 5.094066] systemd[1]: Starting Cleanup udev Database... lb # [ 5.069260] systemd[1]: systemd-udevd.service: Consumed 1.480s CPU time over 3.101s wall clock time, 22.3M memory peak. lb # [ 5.070763] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. lb # [ 5.071732] systemd[1]: Closed udev Control Socket. worker1 # [ 5.096458] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. client # [ 5.032557] systemd[1]: Stopped Load Kernel Modules. worker1 # [ 5.100198] systemd[1]: Stopped Create Static Device Nodes in /dev. lb # [ 5.076265] systemd[1]: Starting Cleanup udev Database... client # [ 5.036288] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker1 # [ 5.101087] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. lb # [ 5.077042] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker1 # [ 5.102172] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. lb # [ 5.078090] systemd[1]: Stopped Create Static Device Nodes in /dev. lb # [ 5.078948] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker1 # [ 5.104309] systemd[1]: kmod-static-nodes.service: Deactivated successfully. client # [ 5.040508] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. client # [ 5.043124] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker1 # [ 5.108090] systemd[1]: Stopped Create List of Static Device Nodes. lb # [ 5.084381] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. worker1 # [ 5.108989] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker1 # [ 5.109976] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 5.085403] systemd[1]: kmod-static-nodes.service: Deactivated successfully. lb # [ 5.086359] systemd[1]: Stopped Create List of Static Device Nodes. client # [ 5.048404] systemd[1]: Stopped Create System Files and Directories. client # [ 5.049432] systemd[1]: Stopped target Local File Systems. client # [ 5.050356] systemd[1]: Stopped target Preparation for Local File Systems. client # [ 5.052177] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. client # [ 5.053553] systemd[1]: Stopped Coldplug All udev Devices. client # [ 5.054375] systemd[1]: Stopping Rule-based Manager for Device Events and Files... client # [ 5.055439] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 5.060745] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 5.124659] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. client # [ 5.061832] systemd[1]: initrd-cleanup.service: Deactivated successfully. worker1 # [ 5.127019] systemd[1]: Finished Cleanup udev Database. client # [ 5.064388] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker1 # [ 5.128909] systemd[1]: Reached target Switch Root. worker1 # [ 5.132297] systemd[1]: Starting NixOS Activation... client # [ 5.068194] systemd[1]: systemd-udevd.service: Deactivated successfully. client # [ 5.069201] systemd[1]: Stopped Rule-based Manager for Device Events and Files. lb # [ 5.109460] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. lb # [ 5.112573] systemd[1]: Finished Cleanup udev Database. client # [ 5.072267] systemd[1]: systemd-udevd.service: Consumed 1.389s CPU time over 3.119s wall clock time, 22.3M memory peak. client # [ 5.073753] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. lb # [ 5.115574] systemd[1]: Reached target Switch Root. client # [ 5.076155] systemd[1]: Closed udev Control Socket. lb # [ 5.116383] systemd[1]: Starting NixOS Activation... client # [ 5.080282] systemd[1]: Starting Cleanup udev Database... client # [ 5.081106] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. client # [ 5.082140] systemd[1]: Stopped Create Static Device Nodes in /dev. client # [ 5.082986] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. client # [ 5.088151] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. client # [ 5.089157] systemd[1]: kmod-static-nodes.service: Deactivated successfully. client # [ 5.090105] systemd[1]: Stopped Create List of Static Device Nodes. client # [ 5.090959] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 5.091941] systemd[1]: Finished rw-sysroot-nix-store.service. client # [ 5.112394] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. client # [ 5.114507] systemd[1]: Finished Cleanup udev Database. client # [ 5.116516] systemd[1]: Reached target Switch Root. client # [ 5.120125] systemd[1]: Starting NixOS Activation... worker1 # [ 5.210117] initrd-nixos-activation-start[178]: booting system configuration /nix/store/yms5bvxhm2a2zidkzddvi0xxaqsl6a35-nixos-system-worker1-test lb # [ 5.192259] initrd-nixos-activation-start[179]: booting system configuration /nix/store/qvl0xxkr124pdkzdvanphvcpy7197xra-nixos-system-lb-test worker1 # [ 5.241612] initrd-nixos-activation-start[178]: running activation script... lb # [ 5.223138] initrd-nixos-activation-start[179]: running activation script... client # [ 5.198609] initrd-nixos-activation-start[175]: booting system configuration /nix/store/c7dwh1z1cl7zl3zmm2gm23k4b20rxnfn-nixos-system-client-test client # [ 5.230309] initrd-nixos-activation-start[175]: running activation script... worker1 # [ 5.474043] initrd-nixos-activation-start[201]: setting up /etc... lb # [ 5.459009] initrd-nixos-activation-start[202]: setting up /etc... client # [ 5.451722] initrd-nixos-activation-start[198]: setting up /etc... worker2 # [ 5.475741] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 5.488558] systemd[1]: Detected virtualization qemu. worker2 # [ 5.491701] systemd[1]: Detected architecture arm64. worker2 # [ 5.495730] systemd[1]: Detected first boot. worker2 # [ 5.501434] systemd[1]: Initializing machine ID from random generator. worker1 # [ 5.591128] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker1 # [ 5.594000] systemd[1]: Finished NixOS Activation. worker1 # [ 5.596164] systemd[1]: Starting Switch Root... lb # [ 5.574539] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. lb # [ 5.577320] systemd[1]: Finished NixOS Activation. lb # [ 5.578472] systemd[1]: Starting Switch Root... worker1 # [ 5.615979] systemd[1]: Switching root. lb # [ 5.598869] systemd[1]: Switching root. client # [ 5.567904] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. client # [ 5.570842] systemd[1]: Finished NixOS Activation. client # [ 5.575901] systemd[1]: Starting Switch Root... client # [ 5.594653] systemd[1]: Switching root. lb # [ 5.783120] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker1 # [ 5.811290] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). client # [ 5.781943] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker2 # [ 5.825217] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 6.031074] systemd[1]: Applying preset policy. lb # [ 6.310547] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 6.322658] systemd[1]: Detected virtualization qemu. lb # [ 6.325635] systemd[1]: Detected architecture arm64. worker1 # [ 6.342176] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 6.329120] systemd[1]: Detected first boot. worker1 # [ 6.354586] systemd[1]: Detected virtualization qemu. worker1 # [ 6.357741] systemd[1]: Detected architecture arm64. lb # [ 6.334529] systemd[1]: Initializing machine ID from random generator. worker1 # [ 6.361439] systemd[1]: Detected first boot. worker1 # [ 6.366869] systemd[1]: Initializing machine ID from random generator. worker2 # [ 6.318186] systemd[1]: Populated /etc with preset unit settings. client # [ 6.305960] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 6.318032] systemd[1]: Detected virtualization qemu. client # [ 6.321015] systemd[1]: Detected architecture arm64. client # [ 6.324659] systemd[1]: Detected first boot. client # [ 6.330014] systemd[1]: Initializing machine ID from random generator. worker2 # [ 6.535301] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker2 # [ 6.536912] systemd[1]: Stopped initrd-switch-root.service. worker2 # [ 6.540559] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker2 # [ 6.544518] systemd[1]: Created slice Slice /system/getty. worker2 # [ 6.546559] systemd[1]: Created slice User and Session Slice. worker2 # [ 6.547743] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker2 # [ 6.550417] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker2 # [ 6.552458] systemd[1]: Expecting device /dev/hvc0... worker2 # [ 6.554308] systemd[1]: Expecting device /dev/ttyAMA0... worker2 # [ 6.555322] systemd[1]: Reached target Local Encrypted Volumes. worker2 # [ 6.557584] systemd[1]: Stopped target initrd-fs.target. worker2 # [ 6.559334] systemd[1]: Stopped target initrd-root-fs.target. worker2 # [ 6.561106] systemd[1]: Stopped target initrd-switch-root.target. worker2 # [ 6.562987] systemd[1]: Reached target Virtual Machines and Containers. worker2 # [ 6.564895] systemd[1]: Reached target Path Units. worker2 # [ 6.566637] systemd[1]: Reached target Remote File Systems. worker2 # [ 6.568432] systemd[1]: Reached target Slice Units. worker2 # [ 6.570274] systemd[1]: Reached target Swaps. worker2 # [ 6.573207] systemd[1]: Listening on Query the User Interactively for a Password. worker2 # [ 6.575982] systemd[1]: Listening on Process Core Dump Socket. worker2 # [ 6.578125] systemd[1]: Listening on Credential Encryption/Decryption. worker2 # [ 6.580386] systemd[1]: Listening on Factory Reset Management. worker2 # [ 6.581567] systemd[1]: Listening on Hostname Service Socket. worker2 # [ 6.585251] systemd[1]: Starting Journal Log Access Socket... worker2 # [ 6.587266] systemd[1]: Listening on Journal Audit Socket. worker2 # [ 6.590731] systemd[1]: Listening on Console Output Muting Service Socket. worker2 # [ 6.592228] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker2 # [ 6.594161] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 6.595963] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 6.601234] systemd[1]: Listening on Disk Repartitioning Service Socket. worker2 # [ 6.602586] systemd[1]: Listening on udev Control Socket. worker2 # [ 6.604019] systemd[1]: Listening on udev Varlink Socket. worker2 # [ 6.607254] systemd[1]: Mounting Huge Pages File System... worker2 # [ 6.613987] systemd[1]: Mounting POSIX Message Queue File System... lb # [ 6.648831] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 6.623408] systemd[1]: Mounting Kernel Debug File System... worker2 # [ 6.634179] systemd[1]: Mounting Kernel Trace File System... worker1 # [ 6.689495] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 6.644496] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 6.649547] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 6.657640] systemd[1]: Mounting Kernel Configuration File System... client # [ 6.648513] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 6.657988] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm worker2 # [ 6.658272] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker2 # [ 6.658533] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker2 # [ 6.674544] systemd[1]: Mounting FUSE Control File System... worker2 # [ 6.676136] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker2 # [ 6.705309] systemd[1]: Starting Journal Service... worker2 # [ 6.722242] systemd[1]: Starting Load Kernel Modules... worker2 # [ 6.738617] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker2 # [ 6.752572] systemd[1]: Starting Remount Root and Kernel File Systems... worker2 # [ 6.754712] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 6.769466] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 6.776039] systemd[1]: Listening on Journal Log Access Socket. worker2 # [ 6.779983] systemd[1]: Mounted Huge Pages File System. worker2 # [ 6.783655] systemd[1]: Mounted POSIX Message Queue File System. worker2 # [ 6.786266] systemd[1]: Mounted Kernel Debug File System. worker2 # [ 6.789008] systemd[1]: Mounted Kernel Trace File System. worker2 # [ 6.793770] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 6.794234] systemd[1]: Mounted FUSE Control File System. worker2 # [ 6.812618] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 6.817517] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 6.862425] systemd[1]: Applying preset policy. worker1 # [ 6.913878] systemd[1]: Applying preset policy. worker2 # [ 6.868473] EXT4-fs (vda): re-mounted fbdf700a-8a52-42f9-bc71-8eb921ea0479. worker2 # [ 6.877275] systemd[1]: Finished Remount Root and Kernel File Systems. client # [ 6.871075] systemd[1]: Applying preset policy. worker2 # [ 6.879748] systemd[1]: Listening on Disk Image Download Service Socket. worker2 # [ 6.880097] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 6.887387] systemd[1]: Starting Load/Save OS Random Seed... worker2 # [ 6.890037] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 6.906001] systemd[1]: Finished Load Kernel Modules. worker2 # [ 6.910406] systemd-journald[264]: Collecting audit messages is enabled. worker2 # [ 6.921173] systemd[1]: Starting Firewall... worker2 # [ 6.938052] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 6.949319] systemd[1]: Started Journal Service. worker2 # [ 6.952321] systemd[1]: Queued start job for default target Multi-User System. worker2 # [ 6.953485] systemd[1]: systemd-journald.service: Deactivated successfully. worker2 # [ 6.964616] systemd-modules-load[265]: Module 'atkbd' is built in worker2 # [ 6.972441] systemd-modules-load[265]: Module 'loop' is built in worker2 # [ 6.985088] systemd[1]: Starting Flush Journal to Persistent Storage... worker2 # [ 7.009024] systemd[1]: Finished Load/Save OS Random Seed. worker2 # [ 7.014361] systemd[1]: Reached target First Boot Complete. worker2 # [ 7.032503] systemd-oomd[266]: No swap; memory pressure usage will be degraded worker2 # [ 7.049216] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker2 # [ 7.054758] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 7.064211] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 7.082873] systemd-journald[264]: Received client request to flush runtime journal. lb # [ 7.133375] systemd[1]: Populated /etc with preset unit settings. client # [ 7.109539] systemd[1]: Populated /etc with preset unit settings. worker1 # [ 7.180842] systemd[1]: Populated /etc with preset unit settings. worker2 # [ 7.116752] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 7.124170] systemd[1]: Finished Flush Journal to Persistent Storage. worker2 # [ 7.269219] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 7.276361] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 7.280368] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 7.330002] systemd[1]: initrd-switch-root.service: Deactivated successfully. lb # [ 7.367684] systemd[1]: initrd-switch-root.service: Deactivated successfully. client # [ 7.331257] systemd[1]: Stopped initrd-switch-root.service. lb # [ 7.368932] systemd[1]: Stopped initrd-switch-root.service. client # [ 7.334401] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. lb # [ 7.372154] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. client # [ 7.337449] systemd[1]: Created slice Slice /system/getty. lb # [ 7.375202] systemd[1]: Created slice Slice /system/getty. client # [ 7.339312] systemd[1]: Created slice User and Session Slice. lb # [ 7.377077] systemd[1]: Created slice User and Session Slice. worker1 # [ 7.402903] systemd[1]: initrd-switch-root.service: Deactivated successfully. client # [ 7.340665] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 7.342338] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker1 # [ 7.404199] systemd[1]: Stopped initrd-switch-root.service. lb # [ 7.378499] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 7.344019] systemd[1]: Expecting device /dev/hvc0... lb # [ 7.378895] systemd[1]: Started Forward Password Requests to Wall Directory Watch. lb # [ 7.379182] systemd[1]: Expecting device /dev/hvc0... worker1 # [ 7.407429] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. lb # [ 7.379426] systemd[1]: Expecting device /dev/ttyAMA0... client # [ 7.346203] systemd[1]: Expecting device /dev/ttyAMA0... lb # [ 7.379661] systemd[1]: Reached target Local Encrypted Volumes. lb # [ 7.379898] systemd[1]: Stopped target initrd-fs.target. client # [ 7.347939] systemd[1]: Reached target Local Encrypted Volumes. lb # [ 7.380130] systemd[1]: Stopped target initrd-root-fs.target. worker1 # [ 7.411119] systemd[1]: Created slice Slice /system/getty. lb # [ 7.380354] systemd[1]: Stopped target initrd-switch-root.target. lb # [ 7.380580] systemd[1]: Reached target Virtual Machines and Containers. client # [ 7.349919] systemd[1]: Stopped target initrd-fs.target. lb # [ 7.380817] systemd[1]: Reached target Path Units. worker1 # [ 7.413139] systemd[1]: Created slice User and Session Slice. lb # [ 7.381049] systemd[1]: Reached target Remote File Systems. lb # [ 7.381268] systemd[1]: Reached target Slice Units. client # [ 7.351665] systemd[1]: Stopped target initrd-root-fs.target. lb # [ 7.381486] systemd[1]: Reached target Swaps. client # [ 7.353628] systemd[1]: Stopped target initrd-switch-root.target. lb # [ 7.392096] systemd[1]: Listening on Query the User Interactively for a Password. client # [ 7.354755] systemd[1]: Reached target Virtual Machines and Containers. worker1 # [ 7.414524] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 7.357118] systemd[1]: Reached target Path Units. worker1 # [ 7.414948] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker1 # [ 7.415253] systemd[1]: Expecting device /dev/hvc0... lb # [ 7.395073] systemd[1]: Listening on Process Core Dump Socket. worker1 # [ 7.415504] systemd[1]: Expecting device /dev/ttyAMA0... client # [ 7.358077] systemd[1]: Reached target Remote File Systems. worker1 # [ 7.415755] systemd[1]: Reached target Local Encrypted Volumes. worker1 # [ 7.416007] systemd[1]: Stopped target initrd-fs.target. client # [ 7.359515] systemd[1]: Reached target Slice Units. lb # [ 7.397194] systemd[1]: Listening on Credential Encryption/Decryption. worker1 # [ 7.416247] systemd[1]: Stopped target initrd-root-fs.target. worker1 # [ 7.416484] systemd[1]: Stopped target initrd-switch-root.target. client # [ 7.360996] systemd[1]: Reached target Swaps. worker1 # [ 7.416728] systemd[1]: Reached target Virtual Machines and Containers. lb # [ 7.399588] systemd[1]: Listening on Factory Reset Management. worker1 # [ 7.416976] systemd[1]: Reached target Path Units. worker1 # [ 7.417212] systemd[1]: Reached target Remote File Systems. worker1 # [ 7.417438] systemd[1]: Reached target Slice Units. lb # [ 7.400767] systemd[1]: Listening on Hostname Service Socket. client # [ 7.364328] systemd[1]: Listening on Query the User Interactively for a Password. worker1 # [ 7.417666] systemd[1]: Reached target Swaps. client # [ 7.367280] systemd[1]: Listening on Process Core Dump Socket. worker1 # [ 7.429101] systemd[1]: Listening on Query the User Interactively for a Password. lb # [ 7.404699] systemd[1]: Starting Journal Log Access Socket... client # [ 7.369528] systemd[1]: Listening on Credential Encryption/Decryption. worker1 # [ 7.432245] systemd[1]: Listening on Process Core Dump Socket. lb # [ 7.406887] systemd[1]: Listening on Journal Audit Socket. client # [ 7.371875] systemd[1]: Listening on Factory Reset Management. worker1 # [ 7.434457] systemd[1]: Listening on Credential Encryption/Decryption. client # [ 7.373173] systemd[1]: Listening on Hostname Service Socket. lb # [ 7.410355] systemd[1]: Listening on Console Output Muting Service Socket. worker1 # [ 7.436816] systemd[1]: Listening on Factory Reset Management. lb # [ 7.411864] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker1 # [ 7.438060] systemd[1]: Listening on Hostname Service Socket. client # [ 7.377215] systemd[1]: Starting Journal Log Access Socket... lb # [ 7.413356] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os lb # [ 7.415385] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki client # [ 7.379391] systemd[1]: Listening on Journal Audit Socket. worker1 # [ 7.442708] systemd[1]: Starting Journal Log Access Socket... client # [ 7.382154] systemd[1]: Listening on Console Output Muting Service Socket. worker1 # [ 7.444387] systemd[1]: Listening on Journal Audit Socket. client # [ 7.383700] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. lb # [ 7.421760] systemd[1]: Listening on Disk Repartitioning Service Socket. lb # [ 7.423089] systemd[1]: Listening on udev Control Socket. worker1 # [ 7.448067] systemd[1]: Listening on Console Output Muting Service Socket. client # [ 7.385535] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os lb # [ 7.424572] systemd[1]: Listening on udev Varlink Socket. worker1 # [ 7.449589] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. client # [ 7.388064] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker1 # [ 7.451306] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os lb # [ 7.427904] systemd[1]: Mounting Huge Pages File System... worker1 # [ 7.454571] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki client # [ 7.393699] systemd[1]: Listening on Disk Repartitioning Service Socket. lb # [ 7.430818] systemd[1]: Mounting POSIX Message Queue File System... client # [ 7.395025] systemd[1]: Listening on udev Control Socket. client # [ 7.396551] systemd[1]: Listening on udev Varlink Socket. worker1 # [ 7.460287] systemd[1]: Listening on Disk Repartitioning Service Socket. client # [ 7.399788] systemd[1]: Mounting Huge Pages File System... worker1 # [ 7.461681] systemd[1]: Listening on udev Control Socket. worker1 # [ 7.463218] systemd[1]: Listening on udev Varlink Socket. client # [ 7.405293] systemd[1]: Mounting POSIX Message Queue File System... worker1 # [ 7.467918] systemd[1]: Mounting Huge Pages File System... worker1 # [ 7.471266] systemd[1]: Mounting POSIX Message Queue File System... lb # [ 7.447889] systemd[1]: Mounting Kernel Debug File System... client # [ 7.416199] systemd[1]: Mounting Kernel Debug File System... client # [ 7.420003] systemd[1]: Mounting Kernel Trace File System... lb # [ 7.456487] systemd[1]: Mounting Kernel Trace File System... worker1 # [ 7.482114] systemd[1]: Mounting Kernel Debug File System... lb # [ 7.467839] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 7.491672] systemd[1]: Mounting Kernel Trace File System... client # [ 7.433359] systemd[1]: Starting Create List of Static Device Nodes... lb # [ 7.471139] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 7.435741] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 7.484740] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 7.511175] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 7.511547] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 7.489925] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm client # [ 7.458484] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 7.519282] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 7.519627] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm client # [ 7.460854] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm worker1 # [ 7.519884] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker1 # [ 7.520142] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 7.499676] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore lb # [ 7.500031] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse client # [ 7.467146] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore client # [ 7.467500] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 7.514288] systemd[1]: Mounting FUSE Control File System... worker1 # [ 7.538563] systemd[1]: Mounting FUSE Control File System... worker1 # [ 7.538951] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 7.515824] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 client # [ 7.495624] systemd[1]: Mounting FUSE Control File System... client # [ 7.496907] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker1 # [ 7.562083] systemd[1]: Starting Journal Service... lb # [ 7.550377] systemd[1]: Starting Journal Service... client # [ 7.517115] systemd[1]: Starting Journal Service... lb # [ 7.560568] systemd[1]: Starting Load Kernel Modules... worker1 # [ 7.592563] systemd[1]: Starting Load Kernel Modules... worker2 # [ 7.528222] systemd[1]: Mounting /run/wrappers... client # [ 7.531772] systemd[1]: Starting Load Kernel Modules... worker2 # [ 7.541384] systemd-udevd[302]: Using default interface naming scheme 'v261'. lb # [ 7.584950] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker1 # [ 7.611944] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... client # [ 7.551819] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker1 # [ 7.617210] systemd[1]: Starting Remount Root and Kernel File Systems... worker1 # [ 7.618970] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 7.561502] systemd[1]: Starting Remount Root and Kernel File Systems... worker1 # [ 7.625494] systemd[1]: Starting Coldplug All udev Devices... client # [ 7.561868] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 7.605681] systemd[1]: Starting Remount Root and Kernel File Systems... lb # [ 7.606075] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.632264] systemd[1]: Listening on Journal Log Access Socket. worker1 # [ 7.632769] systemd[1]: Mounted Huge Pages File System. worker1 # [ 7.633164] systemd[1]: Mounted POSIX Message Queue File System. worker1 # [ 7.633536] systemd[1]: Mounted Kernel Debug File System. worker1 # [ 7.633908] systemd[1]: Mounted Kernel Trace File System. lb # [ 7.616790] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 7.642776] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 7.644988] systemd[1]: Mounted FUSE Control File System. worker2 # [ 7.581021] systemd[1]: Mounted /run/wrappers. worker2 # [ 7.581909] systemd[1]: Reached target Local File Systems. client # [ 7.588730] systemd-journald[268]: Collecting audit messages is enabled. lb # [ 7.629514] systemd-journald[272]: Collecting audit messages is enabled. worker2 # [ 7.590458] systemd[1]: Listening on Boot Loader Control Service Socket. worker2 # [ 7.591484] systemd[1]: Starting register-nix-paths.service... client # [ 7.597521] systemd[1]: Starting Coldplug All udev Devices... lb # [ 7.635715] systemd[1]: Listening on Journal Log Access Socket. client # [ 7.600165] systemd[1]: Listening on Journal Log Access Socket. lb # [ 7.636189] systemd[1]: Mounted Huge Pages File System. lb # [ 7.636580] systemd[1]: Mounted POSIX Message Queue File System. lb # [ 7.636947] systemd[1]: Mounted Kernel Debug File System. lb # [ 7.637312] systemd[1]: Mounted Kernel Trace File System. client # [ 7.605297] systemd[1]: Mounted Huge Pages File System. lb # [ 7.643709] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 7.646745] systemd[1]: Mounted Kernel Configuration File System. client # [ 7.611413] systemd[1]: Mounted POSIX Message Queue File System. worker2 # [ 7.608443] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 7.613199] systemd[1]: Mounted Kernel Debug File System. worker1 # [ 7.674493] systemd[1]: Finished Create List of Static Device Nodes. client # [ 7.600997] systemd[1]: Queued start job for default target Multi-User System. lb # [ 7.642016] systemd[1]: Queued start job for default target Multi-User System. client # [ 7.602517] systemd[1]: systemd-journald.service: Deactivated successfully. worker1 # [ 7.679931] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 7.645039] systemd[1]: systemd-journald.service: Deactivated successfully. worker2 # [ 7.618172] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. lb # [ 7.646393] systemd-modules-load[273]: Module 'atkbd' is built in client # [ 7.625673] systemd[1]: Started Journal Service. lb # [ 7.663285] systemd[1]: Started Journal Service. worker2 # [ 7.625357] systemd[1]: Starting Save Transient machine-id to Disk... client # [ 7.613516] systemd-modules-load[269]: Module 'atkbd' is built in worker2 # [ 7.629536] systemd[1]: Starting Create System Files and Directories... lb # [ 7.657467] systemd-modules-load[273]: Module 'loop' is built in client # [ 7.618959] systemd-modules-load[269]: Module 'loop' is built in client # [ 7.621768] systemd[1]: Mounted Kernel Trace File System. client # [ 7.629149] systemd[1]: Finished Create List of Static Device Nodes. client # [ 7.630102] systemd[1]: Mounted Kernel Configuration File System. lb # [ 7.675471] systemd[1]: Finished Load Kernel Modules. client # [ 7.638709] systemd[1]: Finished Load Kernel Modules. lb # [ 7.688116] systemd[1]: Starting Firewall... worker1 # [ 7.728945] systemd[1]: Finished Load Kernel Modules. worker1 # [ 7.733345] systemd[1]: Starting Firewall... lb # [ 7.697338] systemd[1]: Starting Apply Kernel Variables... client # [ 7.659888] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 7.740088] systemd[1]: Starting Apply Kernel Variables... lb # [ 7.705427] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 7.682055] EXT4-fs (vda): re-mounted b8f4d9fb-9828-4d03-a7aa-d8b67cf041ea. lb # [ 7.706500] systemd[1]: Mounted FUSE Control File System. worker1 # [ 7.749769] systemd-journald[271]: Collecting audit messages is enabled. client # [ 7.667439] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 7.730142] EXT4-fs (vda): re-mounted 43eeae1b-fdb2-40d2-be15-b4a34cfa4c0d. lb # [ 7.728414] systemd[1]: Finished Remount Root and Kernel File Systems. client # [ 7.689729] systemd[1]: Mounted FUSE Control File System. client # [ 7.692127] systemd[1]: Finished Remount Root and Kernel File Systems. client # [ 7.693101] systemd[1]: Listening on Disk Image Download Service Socket. lb # [ 7.734374] systemd[1]: Listening on Disk Image Download Service Socket. client # [ 7.697427] systemd[1]: Starting Flush Journal to Persistent Storage... client # [ 7.698407] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker1 # [ 7.781136] systemd[1]: Started Journal Service. worker1 # [ 7.783428] EXT4-fs (vda): re-mounted 4e060669-9984-4396-8f73-f143f0318f33. lb # [ 7.748478] systemd-oomd[274]: No swap; memory pressure usage will be degraded client # [ 7.711889] systemd[1]: Starting Load/Save OS Random Seed... worker2 # [ 7.727676] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker1 # [ 7.781833] systemd[1]: Queued start job for default target Multi-User System. worker1 # [ 7.782977] systemd[1]: systemd-journald.service: Deactivated successfully. lb # [ 7.760098] systemd[1]: Starting Flush Journal to Persistent Storage... client # [ 7.721238] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 7.761670] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 7.738157] systemd[1]: Finished Save Transient machine-id to Disk. lb # [ 7.771228] systemd[1]: Starting Load/Save OS Random Seed... worker1 # [ 7.796736] systemd-modules-load[272]: Module 'atkbd' is built in client # [ 7.731567] systemd-oomd[271]: No swap; memory pressure usage will be degraded lb # [ 7.776148] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.801758] systemd-modules-load[272]: Module 'loop' is built in lb # [ 7.777383] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. client # [ 7.738856] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker1 # [ 7.813223] systemd[1]: Finished Remount Root and Kernel File Systems. worker1 # [ 7.824346] systemd[1]: Listening on Disk Image Download Service Socket. worker1 # [ 7.834680] systemd[1]: Starting Flush Journal to Persistent Storage... worker2 # [ 7.788806] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 7.843863] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore client # [ 7.813412] systemd-journald[268]: Received client request to flush runtime journal. worker1 # [ 7.863271] systemd[1]: Starting Load/Save OS Random Seed... worker1 # [ 7.871151] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 7.870026] systemd-journald[272]: Received client request to flush runtime journal. worker2 # [ 7.857999] systemd[1]: Finished Create System Files and Directories. client # [ 7.848455] systemd[1]: Finished Load/Save OS Random Seed. client # [ 7.849414] systemd[1]: Reached target First Boot Complete. worker2 # [ 7.869516] systemd[1]: Starting Rebuild Journal Catalog... client # [ 7.857714] systemd[1]: Finished Flush Journal to Persistent Storage. worker1 # [ 7.922214] systemd-oomd[274]: No swap; memory pressure usage will be degraded worker1 # [ 7.923568] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 7.880846] systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [ 7.869179] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 7.947611] systemd-journald[271]: Received client request to flush runtime journal. lb # [ 7.916437] systemd[1]: Finished Load/Save OS Random Seed. lb # [ 7.917326] systemd[1]: Reached target First Boot Complete. lb # [ 7.918106] systemd[1]: Finished Apply Kernel Variables. lb # [ 7.924536] systemd[1]: Finished Flush Journal to Persistent Storage. worker1 # [ 7.988302] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker1 # [ 7.989425] systemd[1]: Finished Load/Save OS Random Seed. worker1 # [ 7.991996] systemd[1]: Reached target First Boot Complete. worker1 # [ 7.994317] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 7.995329] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 8.000431] systemd[1]: Finished Flush Journal to Persistent Storage. lb # [ 7.996760] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 8.004824] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 7.996414] systemd[1]: Finished Record System Boot/Shutdown in UTMP. worker1 # [ 8.102891] systemd[1]: Finished Create Static Device Nodes in /dev. worker1 # [ 8.103850] systemd[1]: Reached target Preparation for Local File Systems. worker1 # [ 8.112566] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker2 # [ 8.079830] systemd[1]: Finished Rebuild Journal Catalog. worker2 # [ 8.084367] systemd[1]: Starting Update is Completed... client # [ 8.084768] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 8.092815] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 8.214000] systemd-udevd[311]: Using default interface naming scheme 'v261'. worker2 # [ 8.187161] systemd[1]: Finished Update is Completed. lb # [ 8.304877] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 8.306184] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 8.314562] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 8.361989] systemd[1]: Mounting /run/wrappers... worker1 # [ 8.394860] systemd[1]: Mounting /run/wrappers... worker1 # [ 8.426661] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 8.463509] systemd[1]: Mounted /run/wrappers. worker1 # [ 8.464794] systemd[1]: Reached target Local File Systems. lb # [ 8.440853] systemd[1]: Mounted /run/wrappers. lb # [ 8.441614] systemd[1]: Reached target Local File Systems. lb # [ 8.448200] systemd[1]: Listening on Boot Loader Control Service Socket. client # [ 8.409387] systemd[1]: Finished Coldplug All udev Devices. worker1 # [ 8.467371] systemd[1]: Listening on Boot Loader Control Service Socket. worker1 # [ 8.476201] systemd[1]: Starting register-nix-paths.service... lb # [ 8.452237] systemd[1]: Starting register-nix-paths.service... client # [ 8.417978] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 8.458441] systemd[1]: Starting Create SUID/SGID Wrappers... lb # [ 8.459300] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. client # [ 8.420193] systemd[1]: Reached target Preparation for Local File Systems. client # [ 8.423566] systemd[1]: Mounting /run/wrappers... client # [ 8.426973] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker1 # [ 8.493283] systemd[1]: Starting Create SUID/SGID Wrappers... lb # [ 8.469557] systemd-udevd[312]: Using default interface naming scheme 'v261'. lb # [ 8.470553] systemd[1]: Starting Save Transient machine-id to Disk... worker1 # [ 8.500382] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. lb # [ 8.483543] systemd[1]: Starting Create System Files and Directories... worker1 # [ 8.510876] systemd[1]: Starting Save Transient machine-id to Disk... worker1 # [ 8.513729] systemd[1]: Starting Create System Files and Directories... client # [ 8.482724] systemd[1]: Mounted /run/wrappers. client # [ 8.484155] systemd[1]: Reached target Local File Systems. client # [ 8.488222] systemd[1]: Listening on Boot Loader Control Service Socket. client # [ 8.492129] systemd[1]: Starting register-nix-paths.service... client # [ 8.499350] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 8.500562] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. client # [ 8.518013] systemd[1]: Starting Save Transient machine-id to Disk... client # [ 8.527794] systemd[1]: Starting Create System Files and Directories... client # [ 8.537007] systemd-udevd[299]: Using default interface naming scheme 'v261'. worker1 # [ 8.622110] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. lb # [ 8.602970] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker1 # [ 8.627819] systemd[1]: Finished Save Transient machine-id to Disk. lb # [ 8.613693] systemd[1]: Finished Save Transient machine-id to Disk. worker2 # [ 8.590723] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 8.598881] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 8.596884] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. client # [ 8.608499] systemd[1]: Finished Save Transient machine-id to Disk. lb # [ 8.722528] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 8.758141] systemd[1]: Finished Create System Files and Directories. client # [ 8.706183] systemd[1]: Finished Create System Files and Directories. worker1 # [ 8.772339] systemd[1]: Starting Rebuild Journal Catalog... worker1 # [ 8.773267] systemd[1]: Starting Record System Boot/Shutdown in UTMP... lb # [ 8.749577] systemd[1]: Finished Create System Files and Directories. client # [ 8.719425] systemd[1]: Starting Rebuild Journal Catalog... client # [ 8.725870] systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [ 8.726945] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 8.783455] systemd[1]: Starting Rebuild Journal Catalog... lb # [ 8.790799] systemd[1]: Starting Record System Boot/Shutdown in UTMP... worker2 # [ 8.788417] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 worker1 # [ 8.885883] systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 8.854367] systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 8.889250] systemd[1]: Finished Rebuild Journal Catalog. client # [ 8.898487] systemd[1]: Starting Update is Completed... worker1 # [ 8.965769] systemd[1]: Finished Rebuild Journal Catalog. worker1 # [ 8.973176] systemd[1]: Starting Update is Completed... lb # [ 8.960730] systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 8.968387] systemd[1]: Finished Update is Completed. lb # [ 9.017851] systemd[1]: Finished Rebuild Journal Catalog. lb # [ 9.031646] systemd[1]: Starting Update is Completed... worker1 # [ 9.085624] systemd[1]: Finished Update is Completed. worker2 # [ 9.100807] systemd[1]: Finished Firewall. lb # [ 9.160097] systemd[1]: Finished Update is Completed. lb # [ 9.259545] systemd[1]: Finished Coldplug All udev Devices. client # [ 9.244179] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 9.287144] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker1 # [ 9.494301] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker1 # [ 9.496489] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 9.438003] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. client # [ 9.441115] systemd[1]: Finished Create SUID/SGID Wrappers. worker2 # [ 9.454937] systemd[1]: Finished Coldplug All udev Devices. worker2 # [ 9.490072] systemd[1]: Finished register-nix-paths.service. worker2 # [ 9.491764] systemd[1]: Reached target System Initialization. worker2 # [ 9.496513] systemd[1]: Started Discard unused filesystem blocks once a week. worker2 # [ 9.497638] systemd[1]: Started Daily Cleanup of Temporary Directories. worker2 # [ 9.499516] systemd[1]: Reached target Timer Units. worker2 # [ 9.503221] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 9.494305] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. lb # [ 9.537495] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 9.510819] systemd[1]: Listening on Nix Daemon Socket. worker2 # [ 9.516089] systemd[1]: Listening on Nix worker-protocol over gRPC. worker2 # [ 9.519728] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker2 # [ 9.528431] systemd[1]: Reached target Socket Units. lb # [ 9.561709] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker2 # [ 9.535974] systemd[1]: Reached target Basic System. worker2 # [ 9.547140] systemd[1]: Starting Import lastlog data into lastlog2 database... worker2 # [ 9.555390] systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 9.547567] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 9.566007] systemd[1]: Starting Post-Boot Actions... worker2 # [ 9.570562] systemd[1]: Started Reset console on configuration changes. worker2 # [ 9.580171] systemd[1]: Starting resolvconf update... worker2 # [ 9.588511] systemd[1]: Starting D-Bus System Message Bus... worker1 # [ 9.691573] systemd[1]: Finished Firewall. worker2 # [ 9.671512] systemd[1]: Finished Post-Boot Actions. lb # [ 9.712150] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. lb # [ 9.717716] systemd[1]: Finished Create SUID/SGID Wrappers. worker2 # [ 9.690169] nsncd[484]: Sep 22 16:53:50.333 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 9.705557] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 9.715955] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 9.723524] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 9.729255] systemd[1]: Starting User Login Management... worker2 # [ 9.754423] systemd[1]: Finished Import lastlog data into lastlog2 database. client # [ 9.744001] (udev-worker)[356]: Network interface NamePolicy= disabled on kernel command line. client # [ 9.749804] (udev-worker)[350]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 9.755671] (udev-worker)[350]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 9.773233] dbus-broker-launch[488]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 9.793821] dbus-broker-launch[488]: NSS returned no entry for 'systemd-timesync' worker2 # [ 9.798069] dbus-broker-launch[488]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" lb # [ 9.841700] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker2 # [ 9.824872] systemd[1]: Started D-Bus System Message Bus. lb # [ 9.860309] systemd[1]: Finished register-nix-paths.service. lb # [ 9.863432] systemd[1]: Reached target System Initialization. lb # [ 9.868600] systemd[1]: Started Discard unused filesystem blocks once a week. lb # [ 9.869647] systemd[1]: Started niks3 garbage collection timer. lb # [ 9.870488] systemd[1]: Started Daily Cleanup of Temporary Directories. lb # [ 9.871394] systemd[1]: Reached target Timer Units. lb # [ 9.884992] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 9.888958] systemd[1]: Listening on niks3 server socket. lb # [ 9.892076] systemd[1]: Listening on Nix Daemon Socket. lb # [ 9.896227] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. lb # [ 9.903128] systemd[1]: Reached target Socket Units. worker2 # [ 9.877330] dbus-broker-launch[488]: Ready lb # [ 9.905398] systemd[1]: Reached target Basic System. lb # [ 9.907368] systemd[1]: Starting Import lastlog data into lastlog2 database... lb # [ 9.914129] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 9.933889] systemd[1]: Starting Post-Boot Actions... lb # [ 9.947187] systemd[1]: Started Reset console on configuration changes. worker2 # [ 9.937238] systemd-logind[499]: New seat seat0. worker1 # [ 10.000537] systemd[1]: Finished Coldplug All udev Devices. worker2 # [ 9.948859] systemd[1]: Started User Login Management. worker2 # [ 9.952206] systemd[1]: Starting linger-users.service... lb # [ 9.989031] systemd[1]: Starting resolvconf update... worker2 # [ 9.979140] systemd[1]: Stopped target Host and Network Name Lookups. worker2 # [ 9.982716] systemd[1]: Stopping Host and Network Name Lookups... worker2 # [ 9.993930] systemd[1]: Stopped target User and Group Name Lookups. lb # [ 10.027805] systemd[1]: Started rustfs.service. worker2 # [ 10.005670] systemd[1]: Stopping User and Group Name Lookups... client # [ 9.999529] systemd[1]: Condition check resulted in Virtio network device being skipped. worker2 # [ 10.012853] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... client # [ 10.003412] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker2 # [ 10.019188] systemd[1]: nscd.service: Deactivated successfully. client # [ 10.009113] systemd[1]: Starting Create SUID/SGID Wrappers... worker2 # [ 10.023029] systemd[1]: Stopped Name Service Cache Daemon (nsncd). lb # [ 10.050094] nsncd[455]: Sep 22 16:53:50.697 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 10.011464] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker2 # [ 10.026141] systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 10.017096] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 10.061978] systemd[1]: Starting rustfs-bucket.service... client # [ 10.019263] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore client # [ 10.026942] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 10.029941] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 10.031210] systemd[1]: Finished register-nix-paths.service. lb # [ 10.072960] systemd[1]: Started Name Service Cache Daemon (nsncd). worker1 # [ 10.098348] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 10.068270] systemd[1]: linger-users.service: Deactivated successfully. worker2 # [ 10.071118] systemd[1]: Finished linger-users.service. worker1 # [ 10.123134] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 10.108205] systemd[1]: Finished Post-Boot Actions. lb # [ 10.128541] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 10.112180] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 10.115625] nsncd[553]: Sep 22 16:53:50.757 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 10.122702] systemd[1]: Reached target Host and Network Name Lookups. lb # [ 10.155866] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 10.129102] systemd[1]: Reached target User and Group Name Lookups. lb # [ 10.161181] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 10.139362] systemd[1]: Finished resolvconf update. worker2 # [ 10.141644] systemd[1]: Reached target Preparation for Network. lb # [ 10.170939] systemd[1]: Started backdoor.service. worker2 # [ 10.152198] systemd[1]: Starting DHCP Client... worker2 # [ 10.153431] systemd[1]: Starting Extra networking commands.... client # [ 10.173393] mousedev: PS/2 mouse device common for all mice lb # [ 10.220317] systemd[1]: Starting D-Bus System Message Bus... worker1 # [ 10.249656] systemd[1]: Finished register-nix-paths.service. worker1 # [ 10.251351] systemd[1]: Reached target System Initialization. worker1 # [ 10.253267] systemd[1]: Started Discard unused filesystem blocks once a week. worker1 # [ 10.255275] systemd[1]: Started Daily Cleanup of Temporary Directories. worker1 # [ 10.260929] systemd[1]: Reached target Timer Units. worker1 # [ 10.264644] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 10.243660] systemd[1]: Starting User Login Management... worker1 # [ 10.279646] systemd[1]: Listening on Nix Daemon Socket. worker1 # [ 10.284612] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 10.293226] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. lb # [ 10.270885] systemd[1]: Finished Import lastlog data into lastlog2 database. worker1 # [ 10.304303] systemd[1]: Reached target Socket Units. worker1 # [ 10.311610] systemd[1]: Reached target Basic System. worker1 # [ 10.318757] systemd[1]: Starting Import lastlog data into lastlog2 database... worker1 # [ 10.322445] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 10.328584] systemd[1]: Starting Post-Boot Actions... worker1 # [ 10.336758] systemd[1]: Started Reset console on configuration changes. worker1 # [ 10.347502] systemd[1]: Starting resolvconf update... worker1 # [ 10.356566] systemd[1]: Starting D-Bus System Message Bus... lb # connecting to host... worker2 # [ 10.374385] dhcpcd[586]: dhcpcd-10.3.2 starting worker1 # [ 10.433483] systemd[1]: Finished Post-Boot Actions. worker2 # [ 10.382198] dhcpcd[627]: dev: loaded udev worker2 # [ 10.387147] dhcpcd[627]: no valid interfaces found worker2 # [ 10.390026] dhcpcd[627]: no valid interfaces found worker2 # [ 10.392553] dhcpcd[627]: libudev: received NULL device lb # [ 10.423438] (udev-worker)[415]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 10.396499] dhcpcd[627]: libudev: received NULL device worker1 # [ 10.464853] systemd[1]: Started Name Service Cache Daemon (nsncd). lb # [ 10.439733] (udev-worker)[416]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 10.473133] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 10.428943] systemd[1]: Finished Extra networking commands.. worker1 # [ 10.481971] nsncd[488]: Sep 22 16:53:51.101 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 10.431927] systemd[1]: Reached target Network. lb # [ 10.461744] (udev-worker)[416]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 10.437602] systemd[1]: Starting Permit User Sessions... worker1 # [ 10.500679] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 10.507429] systemd[1]: Starting User Login Management... worker1 # [ 10.534702] systemd[1]: Finished Import lastlog data into lastlog2 database. worker2 # [ 10.491711] systemd[1]: Finished Permit User Sessions. worker1 # [ 10.545139] dbus-broker-launch[492]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 10.496228] systemd[1]: Started Getty on tty1. worker2 # [ 10.503973] systemd[1]: Reached target Login Prompts. worker1 # [ 10.556624] dbus-broker-launch[492]: NSS returned no entry for 'systemd-timesync' worker1 # [ 10.562957] dbus-broker-launch[492]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker1 # [ 10.578654] systemd[1]: Started D-Bus System Message Bus. worker1 # [ 10.595347] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. client # [ 10.562129] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. client # [ 10.565366] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 10.567882] systemd[1]: Reached target System Initialization. worker1 # [ 10.635359] dbus-broker-launch[492]: Ready client # [ 10.572461] systemd[1]: Started Discard unused filesystem blocks once a week. client # [ 10.573526] systemd[1]: Started Daily Cleanup of Temporary Directories. client # [ 10.574459] systemd[1]: Reached target Timer Units. client # [ 10.575174] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 10.583646] systemd[1]: Listening on Nix Daemon Socket. client # [ 10.587025] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. client # [ 10.593460] systemd[1]: Reached target Socket Units. lb # [ 10.633090] dbus-broker-launch[492]: Looking up NSS user entry for 'systemd-timesync'... client # [ 10.594244] systemd[1]: Reached target Basic System. client # [ 10.594975] systemd[1]: Started backdoor.service. client # [ 10.595674] systemd[1]: Starting Import lastlog data into lastlog2 database... lb # [ 10.643488] dbus-broker-launch[492]: NSS returned no entry for 'systemd-timesync' client # [ 10.609539] systemd[1]: Starting nftables firewall... lb # [ 10.650438] dbus-broker-launch[492]: Invalid user-name in /nix/store/4yiprls1sll47jid3xzq0m9nmi1qz2ww-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker1 # [ 10.692120] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. client # [ 10.631869] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 10.668738] systemd[1]: Stopped target Host and Network Name Lookups. worker1 # [ 10.697452] systemd[1]: Started backdoor.service. lb # [ 10.673405] systemd[1]: Stopping Host and Network Name Lookups... lb # [ 10.674265] systemd[1]: Stopped target User and Group Name Lookups. lb # [ 10.675142] systemd[1]: Stopping User and Group Name Lookups... client # [ 10.638961] systemd[1]: Starting Post-Boot Actions... worker1 # [ 10.711572] systemd-logind[509]: New seat seat0. client # [ 10.647793] systemd[1]: Started Reset console on configuration changes. worker1 # [ 10.720391] systemd[1]: Started User Login Management. lb # [ 10.675995] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... lb # [ 10.698119] systemd-logind[495]: New seat seat0. lb # [ 10.699010] systemd[1]: Started User Login Management. lb # [ 10.699781] systemd[1]: nscd.service: Deactivated successfully. worker1 # [ 10.728088] systemd[1]: Starting linger-users.service... lb # [ 10.718336] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker1 # [ 10.745463] systemd[1]: Stopped target Host and Network Name Lookups. client # [ 10.681481] systemd[1]: Starting resolvconf update... worker1 # [ 10.749853] systemd[1]: Stopping Host and Network Name Lookups... lb # [ 10.727271] systemd[1]: Started D-Bus System Message Bus. worker1 # [ 10.759088] systemd[1]: Stopped target User and Group Name Lookups. worker1 # [ 10.764750] systemd[1]: Stopping User and Group Name Lookups... worker1 # [ 10.765651] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker2 # [ 10.714194] systemd[1]: Starting Virtual Console Setup... worker1 # [ 10.766564] systemd[1]: nscd.service: Deactivated successfully. worker1 # [ 10.767385] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker1 # [ 10.784697] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 10.764482] systemd[1]: Starting linger-users.service... lb # [ 10.773220] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 10.750826] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 10.739740] systemd[1]: Starting D-Bus System Message Bus... client # connecting to host... worker2 # [ 10.754624] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 10.760176] systemd[1]: Starting Virtual Console Setup... lb # [ 10.797042] dbus-broker-launch[492]: Ready worker1 # [ 10.838810] systemd[1]: linger-users.service: Deactivated successfully. worker1 # [ 10.845156] systemd[1]: Finished linger-users.service. worker2 # [ 10.798835] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # connecting to host... worker1 # [ 10.878939] systemd[1]: Started Name Service Cache Daemon (nsncd). client: Guest shell says: b'Spawning backdoor root shell...\n' worker1 # [ 10.879898] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 10.889702] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 10.892778] nsncd[558]: Sep 22 16:53:51.517 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 10.925137] systemd[1]: Finished resolvconf update. client # [ 10.871021] nsncd[451]: Sep 22 16:53:51.521 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 10.925930] systemd[1]: Reached target Preparation for Network. lb # [ 10.920423] systemd[1]: Finished resolvconf update. client: connected to guest root shell client # [ 10.879803] systemd[1]: Finished Post-Boot Actions. worker1 # [ 10.935408] systemd[1]: Starting DHCP Client... lb # [ 10.937074] systemd[1]: Starting DHCP Client... client: (connecting took 11.30 seconds) client # [ 10.884284] systemd[1]: Started Name Service Cache Daemon (nsncd). client: (finished: waiting for the VM to finish booting, in 11.30 seconds) client # [ 10.893955] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 10.939618] systemd[1]: Starting Extra networking commands.... client # [ 10.894893] systemd[1]: Reached target User and Group Name Lookups. lb # [ 10.941777] systemd[1]: linger-users.service: Deactivated successfully. lb # [ 10.949985] systemd[1]: Finished linger-users.service. lb # [ 10.972703] nsncd[573]: Sep 22 16:53:51.622 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 10.937452] systemd[1]: Starting User Login Management... lb # [ 10.984200] systemd[1]: Started Name Service Cache Daemon (nsncd). lb # [ 10.985133] systemd[1]: Reached target Host and Network Name Lookups. lb # [ 10.986037] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 10.996669] systemd-vconsole-setup[654]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 11.006394] systemd[1]: Finished Virtual Console Setup. client # [ 11.003801] systemd[1]: Finished Import lastlog data into lastlog2 database. worker1 # [ 11.068181] (udev-worker)[346]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 11.086342] (udev-worker)[345]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 11.098183] (udev-worker)[346]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 11.066944] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker2 # [ 11.099054] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 11.104503] systemd[1]: Started backdoor.service. client # [ 11.124510] dbus-broker-launch[456]: Looking up NSS user entry for 'systemd-timesync'... client # [ 11.152984] dbus-broker-launch[456]: NSS returned no entry for 'systemd-timesync' client # [ 11.154203] dbus-broker-launch[456]: Invalid user-name in /nix/store/6wvgzhp3xj1v1xybf1ixpbqi5gif5fbv-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker1 # [ 11.221910] dhcpcd[595]: dhcpcd-10.3.2 starting lb # [ 11.210747] systemd[1]: Finished Firewall. worker1 # [ 11.235254] dhcpcd[638]: dev: loaded udev worker2 # [ 11.196208] 8021q: 802.1Q VLAN Support v1.8 lb # [ 11.220673] systemd[1]: Reached target Preparation for Network. lb # [ 11.227614] systemd[1]: Starting Extra networking commands.... worker2 # connecting to host... worker1 # [ 11.278047] 8021q: 802.1Q VLAN Support v1.8 lb # [ 11.243821] systemd[1]: Condition check resulted in Virtio network device being skipped. lb # [ 11.259019] systemd-logind[495]: Watching system buttons on /dev/input/event0 (gpio-keys) lb # [ 11.268452] systemd[1]: Starting Address configuration of eth1... client # [ 11.238526] systemd[1]: Stopped target Host and Network Name Lookups. client # [ 11.248896] systemd[1]: Stopping Host and Network Name Lookups... client # [ 11.261367] systemd[1]: Stopped target User and Group Name Lookups. client # [ 11.262246] systemd[1]: Stopping User and Group Name Lookups... client # [ 11.263067] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker1 # [ 11.332468] systemd[1]: Finished Extra networking commands.. worker1 # [ 11.333404] systemd[1]: Reached target Network. lb # [ 11.316489] dhcpcd[621]: dhcpcd-10.3.2 starting worker2 # [ 11.289762] (udev-worker)[643]: Network interface NamePolicy= disabled on kernel command line. client # [ 11.278219] systemd-logind[484]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) client # [ 11.279615] systemd-logind[484]: Watching system buttons on /dev/input/event0 (gpio-keys) worker2 # [ 11.313027] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker1 # [ 11.353879] systemd[1]: Starting Permit User Sessions... worker1 # [ 11.354852] systemd[1]: Condition check resulted in Virtio network device being skipped. worker1 # [ 11.355929] systemd[1]: Starting Address configuration of eth1... client # [ 11.292452] systemd[1]: Started D-Bus System Message Bus. client # [ 11.293282] systemd-logind[484]: New seat seat0. client # [ 11.293990] systemd[1]: nscd.service: Deactivated successfully. client # [ 11.294802] systemd[1]: Stopped Name Service Cache Daemon (nsncd). lb # [ 11.335320] dhcpcd[635]: dev: loaded udev client # [ 11.295639] systemd[1]: Started User Login Management. worker2 # [ 11.307622] (udev-worker)[642]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 11.329739] (udev-worker)[642]: Network interface NamePolicy= disabled on kernel command line. client # [ 11.320207] systemd[1]: Starting linger-users.service... worker1 # [ 11.398944] cfg80211: Loading compiled-in X.509 certificates for regulatory database client # [ 11.328596] dbus-broker-launch[456]: Ready client # [ 11.335125] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 11.424900] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker1 # [ 11.425407] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker2 # [ 11.374814] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker2 # [ 11.375372] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker1 # [ 11.428355] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker1 # [ 11.428678] cfg80211: failed to load regulatory.db worker2 # [ 11.378918] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker2 # [ 11.379274] cfg80211: failed to load regulatory.db lb # [ 11.424538] 8021q: 802.1Q VLAN Support v1.8 worker1 # [ 11.436493] systemd[1]: Finished Permit User Sessions. worker1 # [ 11.443656] systemd[1]: Started Getty on tty1. worker1 # [ 11.448359] systemd[1]: Reached target Login Prompts. lb # [ 11.459438] 8021q: adding VLAN 0 to HW filter on device eth1 worker2 # [ 11.436605] systemd-logind[499]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker1 # [ 11.514389] 8021q: adding VLAN 0 to HW filter on device eth1 client # [ 11.443690] systemd[1]: linger-users.service: Deactivated successfully. lb # [ 11.500055] mousedev: PS/2 mouse device common for all mice client # [ 11.451393] systemd[1]: Finished linger-users.service. worker1 # [ 11.523925] network-addresses-eth1-start[652]: adding address 192.168.1.3/24... done worker1 # [ 11.549283] 8021q: adding VLAN 0 to HW filter on device eth0 client # [ 11.471939] systemd[1]: Finished resolvconf update. lb # [ 11.512882] network-addresses-eth1-start[633]: adding address 192.168.1.2/24... done worker1 # [ 11.541886] dhcpcd[638]: eth0: waiting for carrier worker1 # [ 11.542647] dhcpcd[638]: eth0: waiting for carrier worker1 # [ 11.543304] dhcpcd[638]: eth0: carrier acquired worker1 # [ 11.548882] network-addresses-eth1-start[652]: adding address 2001:db8:1::3/64... done client # [ 11.487635] nsncd[543]: Sep 22 16:53:52.141 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 11.494653] systemd[1]: Starting DHCP Client... client # [ 11.499341] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 11.502776] systemd[1]: Reached target Host and Network Name Lookups. client # [ 11.503749] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 11.569739] dhcpcd[638]: DUID 00:01:00:01:32:45:6f:20:52:54:00:12:34:56 worker1 # [ 11.570747] dhcpcd[638]: eth0: IAID 00:12:34:56 worker1 # [ 11.571419] dhcpcd[638]: eth0: adding address fe80::5054:ff:fe12:3456 worker2 # [ 11.523253] systemd[1]: Condition check resulted in Virtio network device being skipped. worker1 # [ 11.579798] systemd[1]: Finished Address configuration of eth1. worker2 # [ 11.532287] systemd[1]: Starting Address configuration of eth1... lb # [ 11.570387] network-addresses-eth1-start[633]: adding address 2001:db8:1::2/64... done worker2 # [ 11.593949] 8021q: adding VLAN 0 to HW filter on device eth0 worker2 # [ 11.586181] dhcpcd[627]: eth0: waiting for carrier worker2 # [ 11.587005] dhcpcd[627]: eth0: carrier acquired worker2 # [ 11.605372] dhcpcd[627]: DUID 00:01:00:01:32:45:6f:20:52:54:00:12:34:56 worker2 # [ 11.606401] dhcpcd[627]: eth0: IAID 00:12:34:56 worker2 # [ 11.607045] dhcpcd[627]: eth0: adding address fe80::5054:ff:fe12:3456 lb # [ 11.641856] systemd[1]: Finished Address configuration of eth1. lb # [ 11.659535] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker2 # [ 11.639379] 8021q: adding VLAN 0 to HW filter on device eth1 worker2 # [ 11.641571] network-addresses-eth1-start[676]: adding address 192.168.1.4/24... done worker2 # [ 11.652442] network-addresses-eth1-start[676]: adding address 2001:db8:1::4/64... done lb # [ 11.699837] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' lb # [ 11.700353] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' lb # [ 11.705132] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 lb # [ 11.705469] cfg80211: failed to load regulatory.db worker2 # [ 11.671630] systemd[1]: Finished Address configuration of eth1. worker1 # [ 11.728615] systemd-logind[509]: Watching system buttons on /dev/input/event0 (gpio-keys) worker2 # [ 11.698780] systemd-logind[499]: Watching system buttons on /dev/input/event0 (gpio-keys) lb # [ 11.753203] systemd[1]: Finished Extra networking commands.. client # [ 11.716289] dhcpcd[576]: dhcpcd-10.3.2 starting lb # [ 11.767568] systemd[1]: Reached target Network. client # [ 11.730582] dhcpcd[594]: dev: loaded udev lb # [ 11.776805] systemd[1]: Starting PostgreSQL Server... worker2 # [ 11.773757] mousedev: PS/2 mouse device common for all mice lb # [ 11.808704] systemd[1]: Starting Permit User Sessions... client # [ 11.785311] 8021q: 802.1Q VLAN Support v1.8 lb # [ 11.883929] 8021q: adding VLAN 0 to HW filter on device eth0 lb # [ 11.886911] dhcpcd[635]: eth0: waiting for carrier lb # [ 11.887710] dhcpcd[635]: libudev: received NULL device worker1 # [ 11.932680] mousedev: PS/2 mouse device common for all mice worker2 # [ 11.868189] systemd-logind[499]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) lb # [ 11.899574] dhcpcd[635]: libudev: received NULL device worker2 # [ 11.872575] dhcpcd[627]: eth0: soliciting a DHCP lease lb # [ 11.901801] dhcpcd[635]: eth0: carrier acquired worker2 # [ 11.881711] dhcpcd[627]: eth0: offered 10.0.2.15 from 10.0.2.2 worker2 # [ 11.889218] dhcpcd[627]: eth0: probing address 10.0.2.15/24 client # [ 11.909208] cfg80211: Loading compiled-in X.509 certificates for regulatory database lb # [ 11.935528] systemd[1]: Finished Permit User Sessions. lb # [ 11.937006] dhcpcd[635]: DUID 00:01:00:01:32:45:6f:20:52:54:00:12:34:56 lb # [ 11.937922] dhcpcd[635]: eth0: IAID 00:12:34:56 lb # [ 11.938542] dhcpcd[635]: eth0: adding address fe80::5054:ff:fe12:3456 lb # [ 11.951412] systemd[1]: Started Getty on tty1. lb # [ 11.954912] systemd[1]: Reached target Login Prompts. client # [ 11.940386] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' client # [ 11.942976] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker1 # [ 11.992136] systemd-logind[509]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) client # [ 11.945605] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 client # [ 11.945943] cfg80211: failed to load regulatory.db lb # [ 12.041796] systemd-logind[495]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) client # [ 12.043685] 8021q: adding VLAN 0 to HW filter on device eth0 client # [ 12.030600] dhcpcd[594]: eth0: waiting for carrier client # [ 12.032310] dhcpcd[594]: libudev: received NULL device client # [ 12.035603] dhcpcd[594]: libudev: received NULL device client # [ 12.039521] dhcpcd[594]: eth0: carrier acquired client # [ 12.050975] dhcpcd[594]: DUID 00:01:00:01:32:45:6f:20:52:54:00:12:34:56 client # [ 12.053279] dhcpcd[594]: eth0: IAID 00:12:34:56 client # [ 12.054144] dhcpcd[594]: eth0: adding address fe80::5054:ff:fe12:3456 client # [ 12.061896] systemd[1]: Finished nftables firewall. client # [ 12.066042] systemd[1]: Reached target Preparation for Network. client # [ 12.072214] systemd[1]: Starting Address configuration of eth1... client # [ 12.086986] systemd[1]: Starting Extra networking commands.... client # [ 12.190919] 8021q: adding VLAN 0 to HW filter on device eth1 lb # [ 12.242251] postgresql-pre-start[727]: The files belonging to this database system will be owned by user "postgres". lb # [ 12.243729] postgresql-pre-start[727]: This user must also own the server process. client # [ 12.203957] network-addresses-eth1-start[620]: adding address 192.168.1.1/24... done lb # [ 12.262263] postgresql-pre-start[727]: The database cluster will be initialized with locale "en_US.UTF-8". lb # [ 12.264943] postgresql-pre-start[727]: The default database encoding has accordingly been set to "UTF8". lb # [ 12.266168] postgresql-pre-start[727]: The default text search configuration will be set to "english". lb # [ 12.267533] postgresql-pre-start[727]: Data page checksums are enabled. lb # [ 12.272692] postgresql-pre-start[727]: fixing permissions on existing directory /var/lib/postgresql/18 ... ok lb # [ 12.273977] postgresql-pre-start[727]: creating subdirectories ... ok lb # [ 12.274812] postgresql-pre-start[727]: selecting dynamic shared memory implementation ... posix client # [ 12.236244] network-addresses-eth1-start[620]: adding address 2001:db8:1::1/64... done client # [ 12.269706] systemd[1]: Finished Address configuration of eth1. client # [ 12.344860] dhcpcd[594]: eth0: soliciting a DHCP lease client # [ 12.353025] dhcpcd[594]: eth0: offered 10.0.2.15 from 10.0.2.2 client # [ 12.360269] dhcpcd[594]: eth0: probing address 10.0.2.15/24 client # [ 12.366156] systemd[1]: Finished Extra networking commands.. client # [ 12.369236] systemd[1]: Reached target Network. lb # [ 12.411207] postgresql-pre-start[727]: selecting default "max_connections" ... 100 client # [ 12.380813] systemd[1]: Starting Permit User Sessions... client: (finished: waiting for unit network-addresses-eth1.service, in 12.83 seconds) client: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; } client # [ 12.438037] systemd[1]: Finished Permit User Sessions. client # [ 12.447041] systemd[1]: Started Getty on tty1. client # [ 12.451558] systemd[1]: Reached target Login Prompts. client: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; }, in 0.05 seconds) lb: waiting for unit network-addresses-eth1.service lb: waiting for the VM to finish booting lb: Guest shell says: b'Spawning backdoor root shell...\n' lb: connected to guest root shell lb: (connecting took 0.00 seconds) lb: (finished: waiting for the VM to finish booting, in 0.00 seconds) lb # [ 12.527901] postgresql-pre-start[727]: selecting default "shared_buffers" ... 128MB worker1 # [ 12.585282] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 lb: (finished: waiting for unit network-addresses-eth1.service, in 0.08 seconds) lb: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; } lb: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; }, in 0.05 seconds) worker1: waiting for unit network-addresses-eth1.service worker1: waiting for the VM to finish booting worker1: Guest shell says: b'Spawning backdoor root shell...\n' worker1: connected to guest root shell worker1: (connecting took 0.00 seconds) worker1: (finished: waiting for the VM to finish booting, in 0.00 seconds) client # [ 12.651535] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 worker1: (finished: waiting for unit network-addresses-eth1.service, in 0.14 seconds) worker1: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; } worker1: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; }, in 0.04 seconds) worker2: waiting for unit network-addresses-eth1.service worker2: waiting for the VM to finish booting worker2: Guest shell says: b'Spawning backdoor root shell...\n' worker2: connected to guest root shell worker2: (connecting took 0.00 seconds) worker2: (finished: waiting for the VM to finish booting, in 0.00 seconds) worker2: (finished: waiting for unit network-addresses-eth1.service, in 0.08 seconds) worker2: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; } worker1 # [ 12.945009] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker1 # [ 12.953878] systemd[1]: Starting Virtual Console Setup... worker2: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; }, in 0.04 seconds) lb: waiting for unit niks3.service worker1 # [ 12.972465] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 12.973556] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 12.979760] systemd[1]: Starting Virtual Console Setup... client # [ 12.937812] systemd[1]: Starting Virtual Console Setup... client # [ 12.962696] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 12.969061] systemd[1]: Stopped Virtual Console Setup. client # [ 12.975154] systemd[1]: Starting Virtual Console Setup... worker2 # [ 12.989743] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker2 # [ 12.994583] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 12.995651] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 13.051509] systemd-logind[509]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker2 # [ 13.004432] systemd[1]: Stopping Virtual Console Setup... worker2 # [ 13.005237] systemd[1]: Starting Virtual Console Setup... client # [ 13.001513] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker2 # [ 13.018988] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 13.027650] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 13.028553] systemd[1]: Starting Virtual Console Setup... client # [ 13.046593] systemd-logind[484]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker1 # [ 13.123150] systemd-vconsole-setup[702]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 13.128197] systemd[1]: Finished Virtual Console Setup. worker1 # [ 13.162355] dhcpcd[638]: eth0: soliciting a DHCP lease worker1 # [ 13.168595] dhcpcd[638]: eth0: offered 10.0.2.15 from 10.0.2.2 worker1 # [ 13.172193] dhcpcd[638]: eth0: probing address 10.0.2.15/24 worker2 # [ 13.126306] systemd-vconsole-setup[706]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 13.129908] systemd[1]: Finished Virtual Console Setup. client # [ 13.133803] systemd-vconsole-setup[698]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 13.137640] systemd[1]: Finished Virtual Console Setup. lb # [ 13.217515] dhcpcd[635]: eth0: soliciting a DHCP lease lb # [ 13.224506] dhcpcd[635]: eth0: offered 10.0.2.15 from 10.0.2.2 lb # [ 13.232185] dhcpcd[635]: eth0: probing address 10.0.2.15/24 lb # [ 13.410287] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 lb # [ 13.499695] dhcpcd[635]: eth0: soliciting an IPv6 router lb # [ 13.501196] dhcpcd[635]: eth0: Router Advertisement from fe80::2 lb # [ 13.501998] dhcpcd[635]: eth0: adding address fec0::5054:ff:fe12:3456/64 lb # [ 13.502847] dhcpcd[635]: eth0: adding route to fec0::/64 lb # [ 13.503554] dhcpcd[635]: eth0: adding default route via fe80::2 worker2 # [ 13.545536] dhcpcd[627]: eth0: soliciting an IPv6 router worker2 # [ 13.547880] dhcpcd[627]: eth0: Router Advertisement from fe80::2 worker2 # [ 13.550595] dhcpcd[627]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker2 # [ 13.553410] dhcpcd[627]: eth0: adding route to fec0::/64 worker2 # [ 13.555588] dhcpcd[627]: eth0: adding default route via fe80::2 client # [ 13.645347] dhcpcd[594]: eth0: soliciting an IPv6 router client # [ 13.648934] dhcpcd[594]: eth0: Router Advertisement from fe80::2 client # [ 13.651439] dhcpcd[594]: eth0: adding address fec0::5054:ff:fe12:3456/64 client # [ 13.654320] dhcpcd[594]: eth0: adding route to fec0::/64 client # [ 13.656616] dhcpcd[594]: eth0: adding default route via fe80::2 worker1 # [ 13.739651] dhcpcd[638]: eth0: soliciting an IPv6 router worker1 # [ 13.743021] dhcpcd[638]: eth0: Router Advertisement from fe80::2 worker1 # [ 13.746039] dhcpcd[638]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker1 # [ 13.748963] dhcpcd[638]: eth0: adding route to fec0::/64 worker1 # [ 13.751148] dhcpcd[638]: eth0: adding default route via fe80::2 lb # [ 13.732565] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. lb # [ 13.740062] systemd[1]: Starting Virtual Console Setup... lb # [ 13.759770] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 13.763827] systemd[1]: Stopped Virtual Console Setup. lb # [ 13.768664] systemd[1]: Starting Virtual Console Setup... lb # [ 13.818221] systemd-logind[495]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) lb # [ 14.051004] systemd-vconsole-setup[758]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 14.054574] systemd[1]: Finished Virtual Console Setup. lb # [ 14.215467] postgresql-pre-start[727]: selecting default time zone ... UTC lb # [ 14.218213] postgresql-pre-start[727]: creating configuration files ... ok lb # [ 14.433166] postgresql-pre-start[727]: running bootstrap script ... ok lb # [ 14.933606] postgresql-pre-start[727]: performing post-bootstrap initialization ... ok lb # [ 15.111231] postgresql-pre-start[727]: syncing data to disk ... ok lb # [ 15.112393] postgresql-pre-start[727]: initdb: warning: enabling "trust" authentication for local connections lb # [ 15.113642] postgresql-pre-start[727]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. lb # [ 15.115599] postgresql-pre-start[727]: Success. You can now start the database server using: lb # [ 15.116981] postgresql-pre-start[727]: pg_ctl -D /var/lib/postgresql/18 -l logfile start lb # [ 15.212210] postgres[783]: [783] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit lb # [ 15.217752] postgres[783]: [783] LOG: listening on IPv6 address "::1", port 5432 lb # [ 15.220400] postgres[783]: [783] LOG: listening on IPv4 address "127.0.0.1", port 5432 lb # [ 15.222845] postgres[783]: [783] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432" lb # [ 15.240456] postgres[796]: [796] LOG: database system was shut down at 2026-09-22 16:53:55 GMT lb # [ 15.248079] postgres[783]: [783] LOG: database system is ready to accept connections lb # [ 15.255280] systemd[1]: Started PostgreSQL Server. lb # [ 15.264871] systemd[1]: Starting PostgreSQL Setup Scripts... lb # [ 15.434653] postgresql-setup-start[812]: CREATE DATABASE lb # [ 15.472492] postgresql-setup-start[817]: CREATE ROLE lb # [ 15.486313] postgresql-setup-start[819]: ALTER DATABASE lb # [ 15.491538] systemd[1]: Finished PostgreSQL Setup Scripts. lb # [ 15.493294] systemd[1]: Reached target PostgreSQL. worker2 # [ 17.076502] dhcpcd[627]: eth0: leased 10.0.2.15 for 86400 seconds worker2 # [ 17.079597] dhcpcd[627]: eth0: adding route to 10.0.2.0/24 worker2 # [ 17.086234] dhcpcd[627]: eth0: adding default route via 10.0.2.2 worker2 # [ 17.218666] systemd[1]: Started DHCP Client. worker2 # [ 17.221115] systemd[1]: Reached target Multi-User System. worker2 # [ 17.222099] systemd[1]: Startup finished in 917ms (kernel) + 4.123s (initrd) + 12.179s (userspace) = 17.220s. client # [ 17.749019] dhcpcd[594]: eth0: leased 10.0.2.15 for 86400 seconds client # [ 17.752488] dhcpcd[594]: eth0: adding route to 10.0.2.0/24 client # [ 17.754754] dhcpcd[594]: eth0: adding default route via 10.0.2.2 client # [ 17.901734] systemd[1]: Started DHCP Client. client # [ 17.903808] systemd[1]: Reached target Multi-User System. client # [ 17.905253] systemd[1]: Startup finished in 910ms (kernel) + 4.960s (initrd) + 12.033s (userspace) = 17.904s. lb # [ 18.281080] dhcpcd[635]: eth0: leased 10.0.2.15 for 86400 seconds lb # [ 18.281418] dhcpcd[635]: eth0: adding route to 10.0.2.0/24 lb # [ 18.281553] dhcpcd[635]: eth0: adding default route via 10.0.2.2 worker1 # [ 18.364438] dhcpcd[638]: eth0: leased 10.0.2.15 for 86400 seconds worker1 # [ 18.367588] dhcpcd[638]: eth0: adding route to 10.0.2.0/24 worker1 # [ 18.374536] dhcpcd[638]: eth0: adding default route via 10.0.2.2 lb # [ 18.438933] systemd[1]: Started DHCP Client. lb # [ 18.442227] systemd[1]: Reached target Network is Online. lb # [ 18.447204] systemd[1]: Started Envoy reverse proxy. lb # [ 18.455401] systemd[1]: Started mock-oidc.service. worker1 # [ 18.507556] systemd[1]: Started DHCP Client. worker1 # [ 18.510231] systemd[1]: Reached target Multi-User System. worker1 # [ 18.511238] systemd[1]: Startup finished in 930ms (kernel) + 4.972s (initrd) + 12.606s (userspace) = 18.509s. lb # [ 18.766620] mock-oidc-server[896]: Mock OIDC Server running lb # [ 18.768326] mock-oidc-server[896]: OIDC Address: 192.168.1.2:8080 lb # [ 18.770034] mock-oidc-server[896]: Issue Address: 0.0.0.0:8081 lb # [ 18.771266] mock-oidc-server[896]: Issuer: http://192.168.1.2:8080/oidc lb # [ 18.772554] mock-oidc-server[896]: JWKS: http://192.168.1.2:8080/oidc/.well-known/jwks.json lb # [ 18.773973] mock-oidc-server[896]: Discovery: http://192.168.1.2:8080/oidc/.well-known/openid-configuration lb # [ 18.775591] mock-oidc-server[896]: Issue tokens: http://0.0.0.0:8081/issue?sub=... lb # [ 18.909672] envoy[895]: [2026-09-22 16:53:59.561][895][info][main] [source/server/server.cc:467] initializing epoch 0 (base id=0, hot restart version=11.120) lb # [ 18.912502] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:469] statically linked extensions: lb # [ 18.913956] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:471] envoy.http.ext_proc.response_processors: envoy.extensions.http.ext_proc.save_processing_response lb # [ 18.916351] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:471] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls lb # [ 18.944572] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:471] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls lb # [ 18.949577] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:471] envoy.formatter: envoy.formatter.cel, envoy.formatter.dynamic_modules, envoy.formatter.file_content, envoy.formatter.generic_secret, envoy.formatter.metadata, envoy.formatter.req_without_query lb # [ 18.953249] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:471] envoy.built_in_formatters: envoy.built_in_formatters.cel, envoy.built_in_formatters.http.default, envoy.built_in_formatters.metadata, envoy.built_in_formatters.stream_info.default, envoy.built_in_formatters.xfcc_value lb # [ 18.957393] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:471] envoy.route_config_update_requester: envoy.route_config_update_requester.default lb # [ 18.959692] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:471] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8 lb # [ 18.962800] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:471] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo, envoy.network.dns_resolver.hickory lb # [ 18.965688] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:471] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.fault, envoy.filters.http.a2a, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.ai_protocol_manager, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.api_key_auth, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.bandwidth_share, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cache_v2, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_server, envoy.filters.http.file_system_buffer, envoy.filters.http.filter_chain, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_reverse_transcoder, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.mcp, envoy.filters.http.mcp_json_rest_bridge, envoy.filters.http.mcp_router, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.proto_api_scrubber, envoy.filters.http.proto_message_extraction, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.sse_to_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.thrift_to_metadata, envoy.filters.http.transform, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router lb # [ 18.997605] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:471] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor lb # [ 19.000399] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:471] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.network_namespace, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san lb # [ 19.007604] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:471] envoy.filters.upstream_network: envoy.filters.upstream_network.reverse_tunnel_lifecycle lb # [ 19.010281] envoy[895]: [2026-09-22 16:53:59.563][895][info][main] [source/server/server.cc:471] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter lb # [ 19.012283] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port lb # [ 19.014024] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory lb # [ 19.017273] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.grpc_credentials: envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata lb # [ 19.019404] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.quic.connection_debug_visitor: envoy.quic.connection_debug_visitor.basic, envoy.quic.connection_debug_visitor.quic_stats lb # [ 19.022067] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.load_balancing_policies: envoy.load_balancing_policies.client_side_weighted_round_robin, envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.dynamic_modules, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.load_aware_locality, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.override_host, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset, envoy.load_balancing_policies.wrr_locality lb # [ 19.031344] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.matching.action: envoy.matching.action.transport_socket.name, envoy.matching.actions.format_string, filter-chain-name lb # [ 19.034016] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.generic_proxy.filters: envoy.filters.generic.router lb # [ 19.035726] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.dynamic_modules, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd lb # [ 19.041049] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.matching.stat_tag_matching_data.input: stat_tag_value_input lb # [ 19.042831] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.http.cache_v2: envoy.extensions.http.cache_v2.file_system_http_cache, envoy.extensions.http.cache_v2.simple lb # [ 19.045717] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.tls.certificate_mappers: envoy.tls.certificate_mappers.sni, envoy.tls.certificate_mappers.static_name lb # [ 19.047945] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_mappers: envoy.tls.certificate_mappers.static_name, envoy.tls.upstream_certificate_mappers.filter_state_override lb # [ 19.050845] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.connection_handler: envoy.connection_handler.default lb # [ 19.052709] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.http.forward_client_cert_details: forward_client_cert lb # [ 19.054449] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.envelope, envoy.http.stateful_session.header lb # [ 19.057182] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts lb # [ 19.059987] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche lb # [ 19.062104] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.access_loggers: envoy.access_loggers.dynamic_modules, envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stats, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log lb # [ 19.069093] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.filters.listener: envoy.filters.listener.dynamic_modules, envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.set_filter_state, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector lb # [ 19.075698] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.retry_priorities: envoy.retry_priorities.previous_priorities lb # [ 19.077950] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.clusters: envoy.cluster.dns, envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.composite, envoy.clusters.dynamic_forward_proxy, envoy.clusters.dynamic_modules, envoy.clusters.mcp_multicluster, envoy.clusters.redis, envoy.clusters.reverse_connection lb # [ 19.083183] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation lb # [ 19.085667] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.resource_monitors: envoy.resource_monitors.cgroup_memory, envoy.resource_monitors.cpu_utilization, envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource lb # [ 19.089975] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.regex_engines: envoy.regex_engines.google_re2 lb # [ 19.091630] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.credential_injector, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec, envoy.filters.http.upstream_rbac, envoy.filters.http.wasm lb # [ 19.099152] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.matching.transport_socket.input: envoy.matching.inputs.endpoint_metadata, envoy.matching.inputs.locality_metadata, envoy.matching.inputs.transport_socket_filter_state lb # [ 19.102758] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor lb # [ 19.105600] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.tls.certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 19.107541] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel, envoy.access_loggers.extension_filters.process_ratelimit lb # [ 19.110450] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.ext_authz, envoy.filters.udp.session.http_capsule lb # [ 19.113442] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.protocols: dubbo lb # [ 19.114915] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.matching.generic_proxy_request_input.input: envoy.matching.generic_proxy.input.host, envoy.matching.generic_proxy.input.method, envoy.matching.generic_proxy.input.path, envoy.matching.generic_proxy.input.property, envoy.matching.generic_proxy.input.request, envoy.matching.generic_proxy.input.service lb # [ 19.120104] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.hashable_string, envoy.network.application_protocols, envoy.network.ip, envoy.network.network_namespace, envoy.network.transport_socket.http_11_proxy.info, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.ratelimit.hits_addend, envoy.router.debug_config, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port lb # [ 19.128457] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple lb # [ 19.131743] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.quic.client_packet_writer: envoy.quic.packet_writer.default lb # [ 19.134249] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.dynamic_modules, envoy.filters.udp_listener.udp_proxy lb # [ 19.137298] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff lb # [ 19.140350] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.cel, envoy.tracers.opentelemetry.samplers.dynatrace, envoy.tracers.opentelemetry.samplers.parent_based, envoy.tracers.opentelemetry.samplers.trace_id_ratio_based lb # [ 19.147019] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.health_check.event_sinks: envoy.health_check.event_sink.file lb # [ 19.149292] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.rds_factory: envoy.rds_factory.default worker2 # [ 19.122844] systemd[1]: Reached target Network is Online. lb # [ 19.150857] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment, envoy.tracers.opentelemetry.resource_detectors.static_config worker1 # [ 19.179908] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 19.155705] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter lb # [ 19.159292] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator lb # [ 19.162060] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.resolvers: envoy.ip, envoy.resolvers.reverse_connection worker2 # [ 19.133919] systemd[1]: Starting scheduler-token.service... lb # [ 19.163802] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua, envoy.router.cluster_specifier_plugin.matcher lb # [ 19.166506] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.matching.input_matchers: envoy.matching.input_matchers.generic_request_matcher, envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.dynamic_modules, envoy.matching.matchers.ip, envoy.matching.matchers.metadata_matcher, envoy.matching.matchers.runtime_fraction lb # [ 19.171138] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.route.early_data_policy: envoy.route.early_data_policy.default lb # [ 19.173039] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] quic.http_server_connection: quic.http_server_connection.default lb # [ 19.174864] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.bootstrap: envoy.bootstrap.dynamic_modules, envoy.bootstrap.internal_listener, envoy.bootstrap.reverse_tunnel.downstream_socket_interface, envoy.bootstrap.reverse_tunnel.upstream_socket_interface, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface, envoy.extensions.network.socket_interface.sockmap lb # [ 19.179702] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 19.182386] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.content_parsers: envoy.content_parsers.json lb # [ 19.184001] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.thrift_proxy.transports: auto, framed, header, unframed lb # [ 19.185743] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.quic.connection_id_generator: envoy.quic.connection_id_generator.quic_lb, envoy.quic.deterministic_connection_id_generator lb # [ 19.188148] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain lb # [ 19.189912] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.srds_factory: envoy.srds_factory.default lb # [ 19.191502] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme lb # [ 19.194821] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr lb # [ 19.196653] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case lb # [ 19.198843] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router lb # [ 19.200602] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.http.ext_proc.processing_request_modifiers: envoy.extensions.http.ext_proc.mapped_attribute_builder lb # [ 19.202767] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest lb # [ 19.208093] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso lb # [ 19.210063] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.common.key_value: envoy.key_value.file_based lb # [ 19.211675] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.tracers: envoy.tracers.datadog, envoy.tracers.dynamic_modules, envoy.tracers.fluentd, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin lb # [ 19.215081] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.upstream_options: envoy.extensions.upstreams.http.reverse_tunnel.v3.ReverseTunnelUpstreamCodecOptions, envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options lb # [ 19.219539] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable lb # [ 19.221598] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.http.header_validators: envoy.http.header_validators.envoy_default lb # [ 19.223440] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.dynamic_modules, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.ext_proc, envoy.filters.network.generic_proxy, envoy.filters.network.geoip, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.reverse_tunnel, envoy.filters.network.reverse_tunnel_drain_aware_http_connection_manager, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_bandwidth_limit, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy lb # [ 19.236633] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.serializers: dubbo.hessian2 lb # [ 19.238264] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.dynamic_modules, envoy.tls.cert_validator.spiffe lb # [ 19.241086] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.matching.stat_matching_data.input: stat_full_name_match_input lb # [ 19.243556] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.geoip_providers: envoy.geoip_providers.maxmind lb # [ 19.247678] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.http.injected_credentials: envoy.http.injected_credentials.generic, envoy.http.injected_credentials.oauth2 lb # [ 19.250441] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.generic_proxy.codecs: envoy.generic_proxy.codecs.dubbo, envoy.generic_proxy.codecs.http1 lb # [ 19.252734] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.upstreams: envoy.filters.connection_pools.tcp.generic lb # [ 19.254420] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router worker2 # [ 19.229719] systemd[1]: scheduler-token.service: Deactivated successfully. lb # [ 19.257363] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy lb # [ 19.259976] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret worker2 # [ 19.234687] systemd[1]: Finished scheduler-token.service. lb # [ 19.262150] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.quic.server_preferred_address: quic.server_preferred_address.datasource, quic.server_preferred_address.fixed lb # [ 19.264453] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector, envoy.upstream.local_address_selector.filter_state_override lb # [ 19.267249] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.request_id: envoy.request_id.uuid lb # [ 19.268831] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.health_checkers: envoy.health_checkers.dynamic_modules, envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift worker2 # [ 19.241041] systemd[1]: scheduler-token.service: Consumed 65ms CPU time over 99ms wall clock time, 3.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb # [ 19.272272] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action lb # [ 19.274219] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher worker2 # [ 19.247708] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 19.276031] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation lb # [ 19.277982] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.dynamic_metadata, envoy.matching.inputs.dynamic_module_data_input, envoy.matching.inputs.filter_state, envoy.matching.inputs.local_reply, envoy.matching.inputs.network_namespace, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params lb # [ 19.288700] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.string_matcher: envoy.string_matcher.lua lb # [ 19.291342] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 19.293934] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] network.connection.client: default, envoy_internal lb # [ 19.295551] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.rbac.principals: envoy.rbac.principals.mtls_authenticated lb # [ 19.297419] envoy[895]: [2026-09-22 16:53:59.564][895][info][main] [source/server/server.cc:471] envoy.ssl.server_context_factory: envoy.ssl.server_context_factory.default lb # [ 19.299277] envoy[895]: [2026-09-22 16:53:59.579][895][info][main] [source/server/server.cc:531] HTTP header map info: lb # [ 19.300741] envoy[895]: [2026-09-22 16:53:59.579][895][info][main] [source/server/server.cc:534] request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-host,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-request-id lb # [ 19.315883] envoy[895]: [2026-09-22 16:53:59.579][895][info][main] [source/server/server.cc:534] request trailer map: 120 bytes: lb # [ 19.317457] envoy[895]: [2026-09-22 16:53:59.579][895][info][main] [source/server/server.cc:534] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id lb # [ 19.326172] envoy[895]: [2026-09-22 16:53:59.579][895][info][main] [source/server/server.cc:534] response trailer map: 144 bytes: grpc-message,grpc-status lb # [ 19.327879] envoy[895]: [2026-09-22 16:53:59.609][895][info][main] [source/server/server.cc:968] runtime: {} lb # [ 19.329193] envoy[895]: [2026-09-22 16:53:59.614][895][info][main] [source/server/server.cc:420] Raised file descriptor limits from 1024 to 524288. lb # [ 19.330815] envoy[895]: [2026-09-22 16:53:59.614][895][info][main] [source/server/server.cc:796] Starting admin HTTP server at 127.0.0.1:9901 lb # [ 19.332404] envoy[895]: [2026-09-22 16:53:59.614][895][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:9901 lb # [ 19.333855] envoy[895]: [2026-09-22 16:53:59.617][895][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration lb # [ 19.335394] envoy[895]: [2026-09-22 16:53:59.617][895][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s) lb # [ 19.336985] envoy[895]: [2026-09-22 16:53:59.617][895][info][config] [source/server/configuration_impl.cc:138] loading 2 cluster(s) lb # [ 19.338469] envoy[895]: [2026-09-22 16:53:59.798][895][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s) worker1 # [ 19.363930] nix-grpc-daemon[767]: ts=2026-09-22T16:53:59Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" lb # [ 19.339930] envoy[895]: [2026-09-22 16:53:59.810][895][info][config] [source/server/configuration_impl.cc:164] loading stats configuration lb # [ 19.341590] envoy[895]: [2026-09-22 16:53:59.811][895][warning][main] [source/server/server.cc:1038] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor. lb # [ 19.344774] envoy[895]: [2026-09-22 16:53:59.811][895][info][main] [source/server/server.cc:1077] starting main dispatch loop worker1 # [ 19.372644] nix-grpc-daemon[767]: ts=2026-09-22T16:54:00Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 19.375023] nix-grpc-daemon[767]: ts=2026-09-22T16:54:00Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 19.388993] nix-grpc-daemon[767]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 19.391898] nix-grpc-daemon[767]: E0000 00:00:1790096040.023919 767 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 19.397175] nix-grpc-daemon[767]: E0000 00:00:1790096040.032286 767 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 19.419381] nix-grpc-daemon[767]: ts=2026-09-22T16:54:00Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket lb # [ 19.400193] envoy[895]: [2026-09-22 16:54:00.052][895][info][runtime] [source/common/runtime/runtime_impl.cc:567] RTDS has finished initialization worker1 # [ 19.426687] systemd[1]: Started Nix worker-protocol over gRPC. lb # [ 19.402402] envoy[895]: [2026-09-22 16:54:00.052][895][info][upstream] [source/common/upstream/cluster_manager_impl.cc:250] cm init: all clusters initialized worker1 # [ 19.428202] nix-grpc-daemon[767]: E0000 00:00:1790096040.060604 809 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 19.404830] envoy[895]: [2026-09-22 16:54:00.052][895][info][main] [source/server/server.cc:1057] all clusters initialized. initializing init manager lb # [ 19.409898] envoy[895]: [2026-09-22 16:54:00.061][895][info][config] [source/common/listener_manager/listener_manager_impl.cc:1067] all dependencies initialized. starting workers worker2 # [ 19.400494] nix-grpc-daemon[774]: ts=2026-09-22T16:54:00Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 19.445704] nix-grpc-daemon[767]: E0000 00:00:1790096040.080780 809 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 19.458423] nix-grpc-daemon[767]: E0000 00:00:1790096040.081342 809 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 19.409109] nix-grpc-daemon[774]: ts=2026-09-22T16:54:00Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 19.411458] nix-grpc-daemon[774]: ts=2026-09-22T16:54:00Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 19.467932] nix-grpc-daemon[767]: E0000 00:00:1790096040.101991 809 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 19.422171] nix-grpc-daemon[774]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 19.424155] nix-grpc-daemon[774]: E0000 00:00:1790096040.069030 774 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 19.480313] nix-grpc-daemon[767]: ts=2026-09-22T16:54:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 19.482739] nix-grpc-daemon[767]: ts=2026-09-22T16:54:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 19.431182] nix-grpc-daemon[774]: E0000 00:00:1790096040.078189 774 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 19.451829] nix-grpc-daemon[774]: ts=2026-09-22T16:54:00Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 19.454178] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 19.476252] nix-grpc-daemon[774]: E0000 00:00:1790096040.103655 816 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 19.495780] nix-grpc-daemon[774]: E0000 00:00:1790096040.123148 816 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 19.498841] nix-grpc-daemon[774]: E0000 00:00:1790096040.126243 816 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 19.501598] nix-grpc-daemon[774]: E0000 00:00:1790096040.142034 816 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 19.510322] nix-grpc-daemon[774]: ts=2026-09-22T16:54:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 19.512098] nix-grpc-daemon[774]: ts=2026-09-22T16:54:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 19.691638] nix-grpc-daemon[767]: ts=2026-09-22T16:54:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 19.709681] nix-grpc-daemon[767]: ts=2026-09-22T16:54:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 19.719021] nix-grpc-daemon[774]: ts=2026-09-22T16:54:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 19.735987] nix-grpc-daemon[774]: ts=2026-09-22T16:54:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 19.910127] nix-grpc-daemon[767]: ts=2026-09-22T16:54:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 19.915800] nix-grpc-daemon[767]: ts=2026-09-22T16:54:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 19.935817] nix-grpc-daemon[774]: ts=2026-09-22T16:54:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 19.941241] nix-grpc-daemon[774]: ts=2026-09-22T16:54:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.116782] nix-grpc-daemon[767]: ts=2026-09-22T16:54:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.121878] nix-grpc-daemon[767]: ts=2026-09-22T16:54:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.142136] nix-grpc-daemon[774]: ts=2026-09-22T16:54:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.147244] nix-grpc-daemon[774]: ts=2026-09-22T16:54:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.322730] nix-grpc-daemon[767]: ts=2026-09-22T16:54:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.327856] nix-grpc-daemon[767]: ts=2026-09-22T16:54:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.348394] nix-grpc-daemon[774]: ts=2026-09-22T16:54:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.354133] nix-grpc-daemon[774]: ts=2026-09-22T16:54:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.530425] nix-grpc-daemon[767]: ts=2026-09-22T16:54:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.534568] nix-grpc-daemon[767]: ts=2026-09-22T16:54:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.557757] nix-grpc-daemon[774]: ts=2026-09-22T16:54:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.562249] nix-grpc-daemon[774]: ts=2026-09-22T16:54:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.730917] nix-grpc-daemon[767]: ts=2026-09-22T16:54:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.736238] nix-grpc-daemon[767]: ts=2026-09-22T16:54:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.758324] nix-grpc-daemon[774]: ts=2026-09-22T16:54:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.763421] nix-grpc-daemon[774]: ts=2026-09-22T16:54:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.937040] nix-grpc-daemon[767]: ts=2026-09-22T16:54:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.942005] nix-grpc-daemon[767]: ts=2026-09-22T16:54:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.964710] nix-grpc-daemon[774]: ts=2026-09-22T16:54:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.969847] nix-grpc-daemon[774]: ts=2026-09-22T16:54:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.142777] nix-grpc-daemon[767]: ts=2026-09-22T16:54:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.147915] nix-grpc-daemon[767]: ts=2026-09-22T16:54:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.170824] nix-grpc-daemon[774]: ts=2026-09-22T16:54:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.175838] nix-grpc-daemon[774]: ts=2026-09-22T16:54:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.348925] nix-grpc-daemon[767]: ts=2026-09-22T16:54:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.354036] nix-grpc-daemon[767]: ts=2026-09-22T16:54:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.377040] nix-grpc-daemon[774]: ts=2026-09-22T16:54:02Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.382102] nix-grpc-daemon[774]: ts=2026-09-22T16:54:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.556520] nix-grpc-daemon[767]: ts=2026-09-22T16:54:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.560609] nix-grpc-daemon[767]: ts=2026-09-22T16:54:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.584957] nix-grpc-daemon[774]: ts=2026-09-22T16:54:02Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.588871] nix-grpc-daemon[774]: ts=2026-09-22T16:54:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.756889] nix-grpc-daemon[767]: ts=2026-09-22T16:54:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.761862] nix-grpc-daemon[767]: ts=2026-09-22T16:54:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.785824] nix-grpc-daemon[774]: ts=2026-09-22T16:54:02Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.790952] nix-grpc-daemon[774]: ts=2026-09-22T16:54:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.962749] nix-grpc-daemon[767]: ts=2026-09-22T16:54:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.967686] nix-grpc-daemon[767]: ts=2026-09-22T16:54:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.992212] nix-grpc-daemon[774]: ts=2026-09-22T16:54:02Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.997357] nix-grpc-daemon[774]: ts=2026-09-22T16:54:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.168816] nix-grpc-daemon[767]: ts=2026-09-22T16:54:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.174100] nix-grpc-daemon[767]: ts=2026-09-22T16:54:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.198682] nix-grpc-daemon[774]: ts=2026-09-22T16:54:02Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.203961] nix-grpc-daemon[774]: ts=2026-09-22T16:54:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.374931] nix-grpc-daemon[767]: ts=2026-09-22T16:54:03Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.380432] nix-grpc-daemon[767]: ts=2026-09-22T16:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.405110] nix-grpc-daemon[774]: ts=2026-09-22T16:54:03Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.410253] nix-grpc-daemon[774]: ts=2026-09-22T16:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.583329] nix-grpc-daemon[767]: ts=2026-09-22T16:54:03Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.587678] nix-grpc-daemon[767]: ts=2026-09-22T16:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.613397] nix-grpc-daemon[774]: ts=2026-09-22T16:54:03Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.617622] nix-grpc-daemon[774]: ts=2026-09-22T16:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.783884] nix-grpc-daemon[767]: ts=2026-09-22T16:54:03Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.789624] nix-grpc-daemon[767]: ts=2026-09-22T16:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.814212] nix-grpc-daemon[774]: ts=2026-09-22T16:54:03Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.819511] nix-grpc-daemon[774]: ts=2026-09-22T16:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.990254] nix-grpc-daemon[767]: ts=2026-09-22T16:54:03Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.995660] nix-grpc-daemon[767]: ts=2026-09-22T16:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.020914] nix-grpc-daemon[774]: ts=2026-09-22T16:54:03Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.025914] nix-grpc-daemon[774]: ts=2026-09-22T16:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.196438] nix-grpc-daemon[767]: ts=2026-09-22T16:54:03Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.201490] nix-grpc-daemon[767]: ts=2026-09-22T16:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.226979] nix-grpc-daemon[774]: ts=2026-09-22T16:54:03Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.232217] nix-grpc-daemon[774]: ts=2026-09-22T16:54:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.402330] nix-grpc-daemon[767]: ts=2026-09-22T16:54:04Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.408143] nix-grpc-daemon[767]: ts=2026-09-22T16:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.433386] nix-grpc-daemon[774]: ts=2026-09-22T16:54:04Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.438964] nix-grpc-daemon[774]: ts=2026-09-22T16:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.610569] nix-grpc-daemon[767]: ts=2026-09-22T16:54:04Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.614759] nix-grpc-daemon[767]: ts=2026-09-22T16:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.641972] nix-grpc-daemon[774]: ts=2026-09-22T16:54:04Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.646207] nix-grpc-daemon[774]: ts=2026-09-22T16:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.811250] nix-grpc-daemon[767]: ts=2026-09-22T16:54:04Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.816592] nix-grpc-daemon[767]: ts=2026-09-22T16:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.842627] nix-grpc-daemon[774]: ts=2026-09-22T16:54:04Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.847711] nix-grpc-daemon[774]: ts=2026-09-22T16:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.017351] nix-grpc-daemon[767]: ts=2026-09-22T16:54:04Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.022079] nix-grpc-daemon[767]: ts=2026-09-22T16:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.048819] nix-grpc-daemon[774]: ts=2026-09-22T16:54:04Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.053785] nix-grpc-daemon[774]: ts=2026-09-22T16:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.222852] nix-grpc-daemon[767]: ts=2026-09-22T16:54:04Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.228048] nix-grpc-daemon[767]: ts=2026-09-22T16:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.255009] nix-grpc-daemon[774]: ts=2026-09-22T16:54:04Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.260219] nix-grpc-daemon[774]: ts=2026-09-22T16:54:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.429419] nix-grpc-daemon[767]: ts=2026-09-22T16:54:05Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.434562] nix-grpc-daemon[767]: ts=2026-09-22T16:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.461174] nix-grpc-daemon[774]: ts=2026-09-22T16:54:05Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.467676] nix-grpc-daemon[774]: ts=2026-09-22T16:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.637314] nix-grpc-daemon[767]: ts=2026-09-22T16:54:05Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.641356] nix-grpc-daemon[767]: ts=2026-09-22T16:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.670253] nix-grpc-daemon[774]: ts=2026-09-22T16:54:05Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.674388] nix-grpc-daemon[774]: ts=2026-09-22T16:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.837698] nix-grpc-daemon[767]: ts=2026-09-22T16:54:05Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.843068] nix-grpc-daemon[767]: ts=2026-09-22T16:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.871199] nix-grpc-daemon[774]: ts=2026-09-22T16:54:05Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.876374] nix-grpc-daemon[774]: ts=2026-09-22T16:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.043887] nix-grpc-daemon[767]: ts=2026-09-22T16:54:05Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.049084] nix-grpc-daemon[767]: ts=2026-09-22T16:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.077352] nix-grpc-daemon[774]: ts=2026-09-22T16:54:05Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.082319] nix-grpc-daemon[774]: ts=2026-09-22T16:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.249772] nix-grpc-daemon[767]: ts=2026-09-22T16:54:05Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.254905] nix-grpc-daemon[767]: ts=2026-09-22T16:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.283410] nix-grpc-daemon[774]: ts=2026-09-22T16:54:05Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.288795] nix-grpc-daemon[774]: ts=2026-09-22T16:54:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.455509] nix-grpc-daemon[767]: ts=2026-09-22T16:54:06Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.460746] nix-grpc-daemon[767]: ts=2026-09-22T16:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.489901] nix-grpc-daemon[774]: ts=2026-09-22T16:54:06Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.494961] nix-grpc-daemon[774]: ts=2026-09-22T16:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.663238] nix-grpc-daemon[767]: ts=2026-09-22T16:54:06Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.667438] nix-grpc-daemon[767]: ts=2026-09-22T16:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.697950] nix-grpc-daemon[774]: ts=2026-09-22T16:54:06Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.702099] nix-grpc-daemon[774]: ts=2026-09-22T16:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.863903] nix-grpc-daemon[767]: ts=2026-09-22T16:54:06Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.869079] nix-grpc-daemon[767]: ts=2026-09-22T16:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.898706] nix-grpc-daemon[774]: ts=2026-09-22T16:54:06Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.903692] nix-grpc-daemon[774]: ts=2026-09-22T16:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.069768] nix-grpc-daemon[767]: ts=2026-09-22T16:54:06Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.074811] nix-grpc-daemon[767]: ts=2026-09-22T16:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.104793] nix-grpc-daemon[774]: ts=2026-09-22T16:54:06Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.109865] nix-grpc-daemon[774]: ts=2026-09-22T16:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.275600] nix-grpc-daemon[767]: ts=2026-09-22T16:54:06Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.280849] nix-grpc-daemon[767]: ts=2026-09-22T16:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.311125] nix-grpc-daemon[774]: ts=2026-09-22T16:54:06Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.316394] nix-grpc-daemon[774]: ts=2026-09-22T16:54:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.481478] nix-grpc-daemon[767]: ts=2026-09-22T16:54:07Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.486555] nix-grpc-daemon[767]: ts=2026-09-22T16:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.517300] nix-grpc-daemon[774]: ts=2026-09-22T16:54:07Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.522320] nix-grpc-daemon[774]: ts=2026-09-22T16:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.689142] nix-grpc-daemon[767]: ts=2026-09-22T16:54:07Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.693166] nix-grpc-daemon[767]: ts=2026-09-22T16:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.725085] nix-grpc-daemon[774]: ts=2026-09-22T16:54:07Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.729446] nix-grpc-daemon[774]: ts=2026-09-22T16:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.889784] nix-grpc-daemon[767]: ts=2026-09-22T16:54:07Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.894922] nix-grpc-daemon[767]: ts=2026-09-22T16:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.930339] nix-grpc-daemon[774]: ts=2026-09-22T16:54:07Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.935749] nix-grpc-daemon[774]: ts=2026-09-22T16:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.095829] nix-grpc-daemon[767]: ts=2026-09-22T16:54:07Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.101005] nix-grpc-daemon[767]: ts=2026-09-22T16:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.136960] nix-grpc-daemon[774]: ts=2026-09-22T16:54:07Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.141830] nix-grpc-daemon[774]: ts=2026-09-22T16:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.301702] nix-grpc-daemon[767]: ts=2026-09-22T16:54:07Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.306820] nix-grpc-daemon[767]: ts=2026-09-22T16:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.342966] nix-grpc-daemon[774]: ts=2026-09-22T16:54:07Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.348206] nix-grpc-daemon[774]: ts=2026-09-22T16:54:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.507806] nix-grpc-daemon[767]: ts=2026-09-22T16:54:08Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.512986] nix-grpc-daemon[767]: ts=2026-09-22T16:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.549260] nix-grpc-daemon[774]: ts=2026-09-22T16:54:08Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.554353] nix-grpc-daemon[774]: ts=2026-09-22T16:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.715529] nix-grpc-daemon[767]: ts=2026-09-22T16:54:08Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.719644] nix-grpc-daemon[767]: ts=2026-09-22T16:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.757040] nix-grpc-daemon[774]: ts=2026-09-22T16:54:08Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.761685] nix-grpc-daemon[774]: ts=2026-09-22T16:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.917939] nix-grpc-daemon[767]: ts=2026-09-22T16:54:08Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.922718] nix-grpc-daemon[767]: ts=2026-09-22T16:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.962830] nix-grpc-daemon[774]: ts=2026-09-22T16:54:08Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.968160] nix-grpc-daemon[774]: ts=2026-09-22T16:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.118962] nix-grpc-daemon[767]: ts=2026-09-22T16:54:08Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.124359] nix-grpc-daemon[767]: ts=2026-09-22T16:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.169237] nix-grpc-daemon[774]: ts=2026-09-22T16:54:08Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.174221] nix-grpc-daemon[774]: ts=2026-09-22T16:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.325231] nix-grpc-daemon[767]: ts=2026-09-22T16:54:08Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.330366] nix-grpc-daemon[767]: ts=2026-09-22T16:54:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.375226] nix-grpc-daemon[774]: ts=2026-09-22T16:54:09Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.380430] nix-grpc-daemon[774]: ts=2026-09-22T16:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.530960] nix-grpc-daemon[767]: ts=2026-09-22T16:54:09Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.536232] nix-grpc-daemon[767]: ts=2026-09-22T16:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.581406] nix-grpc-daemon[774]: ts=2026-09-22T16:54:09Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.586397] nix-grpc-daemon[774]: ts=2026-09-22T16:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.738656] nix-grpc-daemon[767]: ts=2026-09-22T16:54:09Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.742566] nix-grpc-daemon[767]: ts=2026-09-22T16:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.789089] nix-grpc-daemon[774]: ts=2026-09-22T16:54:09Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.793739] nix-grpc-daemon[774]: ts=2026-09-22T16:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.939416] nix-grpc-daemon[767]: ts=2026-09-22T16:54:09Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.944636] nix-grpc-daemon[767]: ts=2026-09-22T16:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.994813] nix-grpc-daemon[774]: ts=2026-09-22T16:54:09Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.999855] nix-grpc-daemon[774]: ts=2026-09-22T16:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 29.044829] rustfs-bucket-start[1006]: mb s3://farm lb # [ 29.059257] systemd[1]: Finished rustfs-bucket.service. lb # [ 29.073480] systemd[1]: Starting niks3 server... worker1 # [ 29.145548] nix-grpc-daemon[767]: ts=2026-09-22T16:54:09Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.150337] nix-grpc-daemon[767]: ts=2026-09-22T16:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 29.212081] postgres[1021]: [1021] ERROR: relation "goose_db_version" does not exist at character 36 lb # [ 29.216220] postgres[1021]: [1021] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC worker2 # [ 29.200961] nix-grpc-daemon[774]: ts=2026-09-22T16:54:09Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.205950] nix-grpc-daemon[774]: ts=2026-09-22T16:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 29.239823] niks3-server[1016]: 2026/09/22 16:54:09 OK 20241026095416_initial_model.sql (16.59ms) lb # [ 29.243172] niks3-server[1016]: 2026/09/22 16:54:09 OK 20251210153512_drop_unused_gin_index.sql (3.61ms) lb # [ 29.248389] niks3-server[1016]: 2026/09/22 16:54:09 OK 20251218171726_add_pins.sql (5.16ms) lb # [ 29.252877] niks3-server[1016]: 2026/09/22 16:54:09 OK 20260628120000_add_object_size_and_stats.sql (4.45ms) lb # [ 29.258046] niks3-server[1016]: 2026/09/22 16:54:09 OK 20260905000000_add_claims.sql (5.14ms) lb # [ 29.261634] niks3-server[1016]: 2026/09/22 16:54:09 OK 20260920000000_drop_claims.sql (3.49ms) lb # [ 29.263543] niks3-server[1016]: 2026/09/22 16:54:09 goose: successfully migrated database to version: 20260920000000 lb # [ 29.266880] niks3-server[1016]: 2026/09/22 16:54:09 OK 1_commit_pending_closure.sql (5.22ms) lb # [ 29.269429] niks3-server[1016]: 2026/09/22 16:54:09 OK 2_object_stats_trigger.sql (2.41ms) lb # [ 29.271260] niks3-server[1016]: 2026/09/22 16:54:09 goose: up to current file version: 2 lb # [ 29.273093] niks3-server[1016]: 2026/09/22 16:54:09 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key lb # [ 29.294071] niks3-server[1016]: 2026/09/22 16:54:09 INFO Read proxy enabled — serving cache objects from S3 lb # [ 29.297962] niks3-server[1016]: 2026/09/22 16:54:09 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 29.303003] niks3-server[1016]: 2026/09/22 16:54:09 INFO systemd watchdog enabled interval=15s lb # [ 29.305241] niks3-server[1016]: 2026/09/22 16:54:09 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 29.306681] systemd[1]: Started niks3 server. lb # [ 29.307375] systemd[1]: Reached target Multi-User System. lb # [ 29.308643] systemd[1]: Startup finished in 915ms (kernel) + 4.960s (initrd) + 23.427s (userspace) = 29.302s. worker1 # [ 29.351102] nix-grpc-daemon[767]: ts=2026-09-22T16:54:09Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.356284] nix-grpc-daemon[767]: ts=2026-09-22T16:54:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.407014] nix-grpc-daemon[774]: ts=2026-09-22T16:54:10Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.413281] nix-grpc-daemon[774]: ts=2026-09-22T16:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.556810] nix-grpc-daemon[767]: ts=2026-09-22T16:54:10Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.561894] nix-grpc-daemon[767]: ts=2026-09-22T16:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.613765] nix-grpc-daemon[774]: ts=2026-09-22T16:54:10Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.618876] nix-grpc-daemon[774]: ts=2026-09-22T16:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.764535] nix-grpc-daemon[767]: ts=2026-09-22T16:54:10Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.768588] nix-grpc-daemon[767]: ts=2026-09-22T16:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.822083] nix-grpc-daemon[774]: ts=2026-09-22T16:54:10Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.826191] nix-grpc-daemon[774]: ts=2026-09-22T16:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: waiting for unit niks3.service, in 16.94 seconds) lb: waiting for TCP port 5751 on localhost worker1 # [ 29.965019] nix-grpc-daemon[767]: ts=2026-09-22T16:54:10Z level=info event=scheduler_connected addr=lb:50051 lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded! worker1 # [ 29.969990] nix-grpc-daemon[767]: ts=2026-09-22T16:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: waiting for TCP port 5751 on localhost, in 0.07 seconds) lb: waiting for unit envoy.service lb: (finished: waiting for unit envoy.service, in 0.05 seconds) lb: waiting for TCP port 50051 on localhost lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded! lb # [ 30.027943] envoy[895]: conn peer=[::1]:41740 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=3 lb: (finished: waiting for TCP port 50051 on localhost, in 0.03 seconds) worker2 # [ 30.023082] nix-grpc-daemon[774]: ts=2026-09-22T16:54:10Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.028473] nix-grpc-daemon[774]: ts=2026-09-22T16:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.170826] nix-grpc-daemon[767]: ts=2026-09-22T16:54:10Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.176609] nix-grpc-daemon[767]: ts=2026-09-22T16:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.162669] systemd[1]: Starting scheduler-token.service... worker2 # [ 30.229194] nix-grpc-daemon[774]: ts=2026-09-22T16:54:10Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.232089] nix-grpc-daemon[774]: ts=2026-09-22T16:54:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.241508] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 30.247142] systemd[1]: Finished scheduler-token.service. worker2 # [ 30.249938] systemd[1]: scheduler-token.service: Consumed 43ms CPU time over 82ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 30.377114] nix-grpc-daemon[767]: ts=2026-09-22T16:54:11Z level=info event=scheduler_connected addr=lb:50051 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.08 seconds) subtest: one node schedules and both builders hold a WorkerSession on it lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 30.382612] nix-grpc-daemon[767]: ts=2026-09-22T16:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2 # [ 30.433359] nix-grpc-daemon[774]: ts=2026-09-22T16:54:11Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.438625] nix-grpc-daemon[774]: ts=2026-09-22T16:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.583380] nix-grpc-daemon[767]: ts=2026-09-22T16:54:11Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.588801] nix-grpc-daemon[767]: ts=2026-09-22T16:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.639381] nix-grpc-daemon[774]: ts=2026-09-22T16:54:11Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.644784] nix-grpc-daemon[774]: ts=2026-09-22T16:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.791868] nix-grpc-daemon[767]: ts=2026-09-22T16:54:11Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.796240] nix-grpc-daemon[767]: ts=2026-09-22T16:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.847549] nix-grpc-daemon[774]: ts=2026-09-22T16:54:11Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.851972] nix-grpc-daemon[774]: ts=2026-09-22T16:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.992450] nix-grpc-daemon[767]: ts=2026-09-22T16:54:11Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.997836] nix-grpc-daemon[767]: ts=2026-09-22T16:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.048241] nix-grpc-daemon[774]: ts=2026-09-22T16:54:11Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.053196] nix-grpc-daemon[774]: ts=2026-09-22T16:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.198772] nix-grpc-daemon[767]: ts=2026-09-22T16:54:11Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.209139] nix-grpc-daemon[767]: ts=2026-09-22T16:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.254506] nix-grpc-daemon[774]: ts=2026-09-22T16:54:11Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.264680] nix-grpc-daemon[774]: ts=2026-09-22T16:54:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.405277] nix-grpc-daemon[767]: ts=2026-09-22T16:54:12Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.411137] nix-grpc-daemon[767]: ts=2026-09-22T16:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 31.461296] nix-grpc-daemon[774]: ts=2026-09-22T16:54:12Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.466647] nix-grpc-daemon[774]: ts=2026-09-22T16:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.612286] nix-grpc-daemon[767]: ts=2026-09-22T16:54:12Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.617484] nix-grpc-daemon[767]: ts=2026-09-22T16:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.667664] nix-grpc-daemon[774]: ts=2026-09-22T16:54:12Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.673800] nix-grpc-daemon[774]: ts=2026-09-22T16:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.820162] nix-grpc-daemon[767]: ts=2026-09-22T16:54:12Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.823940] nix-grpc-daemon[767]: ts=2026-09-22T16:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.875977] nix-grpc-daemon[774]: ts=2026-09-22T16:54:12Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.879961] nix-grpc-daemon[774]: ts=2026-09-22T16:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.020474] nix-grpc-daemon[767]: ts=2026-09-22T16:54:12Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.025712] nix-grpc-daemon[767]: ts=2026-09-22T16:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.076912] nix-grpc-daemon[774]: ts=2026-09-22T16:54:12Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.081969] nix-grpc-daemon[774]: ts=2026-09-22T16:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.226586] nix-grpc-daemon[767]: ts=2026-09-22T16:54:12Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.231783] nix-grpc-daemon[767]: ts=2026-09-22T16:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.282951] nix-grpc-daemon[774]: ts=2026-09-22T16:54:12Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.288304] nix-grpc-daemon[774]: ts=2026-09-22T16:54:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.432655] nix-grpc-daemon[767]: ts=2026-09-22T16:54:13Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.438475] nix-grpc-daemon[767]: ts=2026-09-22T16:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 32.489604] nix-grpc-daemon[774]: ts=2026-09-22T16:54:13Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.494999] nix-grpc-daemon[774]: ts=2026-09-22T16:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 32.639158] nix-grpc-daemon[767]: ts=2026-09-22T16:54:13Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.644375] nix-grpc-daemon[767]: ts=2026-09-22T16:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.695651] nix-grpc-daemon[774]: ts=2026-09-22T16:54:13Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.700769] nix-grpc-daemon[774]: ts=2026-09-22T16:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.847121] nix-grpc-daemon[767]: ts=2026-09-22T16:54:13Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.851370] nix-grpc-daemon[767]: ts=2026-09-22T16:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.903681] nix-grpc-daemon[774]: ts=2026-09-22T16:54:13Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.907900] nix-grpc-daemon[774]: ts=2026-09-22T16:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.047483] nix-grpc-daemon[767]: ts=2026-09-22T16:54:13Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.052879] nix-grpc-daemon[767]: ts=2026-09-22T16:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.104405] nix-grpc-daemon[774]: ts=2026-09-22T16:54:13Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.109449] nix-grpc-daemon[774]: ts=2026-09-22T16:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.253616] nix-grpc-daemon[767]: ts=2026-09-22T16:54:13Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.258802] nix-grpc-daemon[767]: ts=2026-09-22T16:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.310499] nix-grpc-daemon[774]: ts=2026-09-22T16:54:13Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.315513] nix-grpc-daemon[774]: ts=2026-09-22T16:54:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.459471] nix-grpc-daemon[767]: ts=2026-09-22T16:54:14Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.464737] nix-grpc-daemon[767]: ts=2026-09-22T16:54:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 33.516555] nix-grpc-daemon[774]: ts=2026-09-22T16:54:14Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.521430] nix-grpc-daemon[774]: ts=2026-09-22T16:54:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 33.665760] nix-grpc-daemon[767]: ts=2026-09-22T16:54:14Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.670980] nix-grpc-daemon[767]: ts=2026-09-22T16:54:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.722445] nix-grpc-daemon[774]: ts=2026-09-22T16:54:14Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.727564] nix-grpc-daemon[774]: ts=2026-09-22T16:54:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.873683] nix-grpc-daemon[767]: ts=2026-09-22T16:54:14Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.877833] nix-grpc-daemon[767]: ts=2026-09-22T16:54:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.930251] nix-grpc-daemon[774]: ts=2026-09-22T16:54:14Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.934269] nix-grpc-daemon[774]: ts=2026-09-22T16:54:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.074163] nix-grpc-daemon[767]: ts=2026-09-22T16:54:14Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.079354] nix-grpc-daemon[767]: ts=2026-09-22T16:54:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.130885] nix-grpc-daemon[774]: ts=2026-09-22T16:54:14Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.135992] nix-grpc-daemon[774]: ts=2026-09-22T16:54:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.280133] nix-grpc-daemon[767]: ts=2026-09-22T16:54:14Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.285136] nix-grpc-daemon[767]: ts=2026-09-22T16:54:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.337051] nix-grpc-daemon[774]: ts=2026-09-22T16:54:14Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.342049] nix-grpc-daemon[774]: ts=2026-09-22T16:54:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.485844] nix-grpc-daemon[767]: ts=2026-09-22T16:54:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.491084] nix-grpc-daemon[767]: ts=2026-09-22T16:54:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.545369] nix-grpc-daemon[774]: ts=2026-09-22T16:54:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.554212] nix-grpc-daemon[774]: ts=2026-09-22T16:54:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 34.691963] nix-grpc-daemon[767]: ts=2026-09-22T16:54:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.697079] nix-grpc-daemon[767]: ts=2026-09-22T16:54:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 34.751699] nix-grpc-daemon[774]: ts=2026-09-22T16:54:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.757227] nix-grpc-daemon[774]: ts=2026-09-22T16:54:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.899707] nix-grpc-daemon[767]: ts=2026-09-22T16:54:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.904235] nix-grpc-daemon[767]: ts=2026-09-22T16:54:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.960004] nix-grpc-daemon[774]: ts=2026-09-22T16:54:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.964161] nix-grpc-daemon[774]: ts=2026-09-22T16:54:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 35.100102] nix-grpc-daemon[767]: ts=2026-09-22T16:54:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 35.105386] nix-grpc-daemon[767]: ts=2026-09-22T16:54:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.160726] nix-grpc-daemon[774]: ts=2026-09-22T16:54:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.166041] nix-grpc-daemon[774]: ts=2026-09-22T16:54:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 35.306341] nix-grpc-daemon[767]: ts=2026-09-22T16:54:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 35.312002] nix-grpc-daemon[767]: ts=2026-09-22T16:54:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.367256] nix-grpc-daemon[774]: ts=2026-09-22T16:54:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.372884] nix-grpc-daemon[774]: ts=2026-09-22T16:54:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 35.512681] nix-grpc-daemon[767]: ts=2026-09-22T16:54:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 35.518060] nix-grpc-daemon[767]: ts=2026-09-22T16:54:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.574005] nix-grpc-daemon[774]: ts=2026-09-22T16:54:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.579371] nix-grpc-daemon[774]: ts=2026-09-22T16:54:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 35.718859] nix-grpc-daemon[767]: ts=2026-09-22T16:54:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 35.724411] nix-grpc-daemon[767]: ts=2026-09-22T16:54:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 35.780858] nix-grpc-daemon[774]: ts=2026-09-22T16:54:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.786165] nix-grpc-daemon[774]: ts=2026-09-22T16:54:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 35.927011] nix-grpc-daemon[767]: ts=2026-09-22T16:54:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 35.931287] nix-grpc-daemon[767]: ts=2026-09-22T16:54:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.988956] nix-grpc-daemon[774]: ts=2026-09-22T16:54:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.993695] nix-grpc-daemon[774]: ts=2026-09-22T16:54:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 36.127966] nix-grpc-daemon[767]: ts=2026-09-22T16:54:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.133284] nix-grpc-daemon[767]: ts=2026-09-22T16:54:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.194790] nix-grpc-daemon[774]: ts=2026-09-22T16:54:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.200185] nix-grpc-daemon[774]: ts=2026-09-22T16:54:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 36.333980] nix-grpc-daemon[767]: ts=2026-09-22T16:54:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.339431] nix-grpc-daemon[767]: ts=2026-09-22T16:54:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.401402] nix-grpc-daemon[774]: ts=2026-09-22T16:54:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.406664] nix-grpc-daemon[774]: ts=2026-09-22T16:54:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 36.540044] nix-grpc-daemon[767]: ts=2026-09-22T16:54:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.544973] nix-grpc-daemon[767]: ts=2026-09-22T16:54:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.607867] nix-grpc-daemon[774]: ts=2026-09-22T16:54:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.613262] nix-grpc-daemon[774]: ts=2026-09-22T16:54:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 36.745654] nix-grpc-daemon[767]: ts=2026-09-22T16:54:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.750807] nix-grpc-daemon[767]: ts=2026-09-22T16:54:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 36.814165] nix-grpc-daemon[774]: ts=2026-09-22T16:54:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.819309] nix-grpc-daemon[774]: ts=2026-09-22T16:54:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 36.953401] nix-grpc-daemon[767]: ts=2026-09-22T16:54:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.957551] nix-grpc-daemon[767]: ts=2026-09-22T16:54:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.022404] nix-grpc-daemon[774]: ts=2026-09-22T16:54:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.026486] nix-grpc-daemon[774]: ts=2026-09-22T16:54:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 37.153903] nix-grpc-daemon[767]: ts=2026-09-22T16:54:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.159034] nix-grpc-daemon[767]: ts=2026-09-22T16:54:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.223116] nix-grpc-daemon[774]: ts=2026-09-22T16:54:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.228841] nix-grpc-daemon[774]: ts=2026-09-22T16:54:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 37.359924] nix-grpc-daemon[767]: ts=2026-09-22T16:54:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.365013] nix-grpc-daemon[767]: ts=2026-09-22T16:54:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.429318] nix-grpc-daemon[774]: ts=2026-09-22T16:54:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.434317] nix-grpc-daemon[774]: ts=2026-09-22T16:54:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 37.565588] nix-grpc-daemon[767]: ts=2026-09-22T16:54:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.570719] nix-grpc-daemon[767]: ts=2026-09-22T16:54:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.635328] nix-grpc-daemon[774]: ts=2026-09-22T16:54:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.640639] nix-grpc-daemon[774]: ts=2026-09-22T16:54:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 37.771530] nix-grpc-daemon[767]: ts=2026-09-22T16:54:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.776784] nix-grpc-daemon[767]: ts=2026-09-22T16:54:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 37.841615] nix-grpc-daemon[774]: ts=2026-09-22T16:54:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.846636] nix-grpc-daemon[774]: ts=2026-09-22T16:54:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 37.979343] nix-grpc-daemon[767]: ts=2026-09-22T16:54:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.983793] nix-grpc-daemon[767]: ts=2026-09-22T16:54:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.049527] nix-grpc-daemon[774]: ts=2026-09-22T16:54:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.053404] nix-grpc-daemon[774]: ts=2026-09-22T16:54:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 38.180072] nix-grpc-daemon[767]: ts=2026-09-22T16:54:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.185262] nix-grpc-daemon[767]: ts=2026-09-22T16:54:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.250387] nix-grpc-daemon[774]: ts=2026-09-22T16:54:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.255462] nix-grpc-daemon[774]: ts=2026-09-22T16:54:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 38.386034] nix-grpc-daemon[767]: ts=2026-09-22T16:54:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.391095] nix-grpc-daemon[767]: ts=2026-09-22T16:54:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.456560] nix-grpc-daemon[774]: ts=2026-09-22T16:54:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.461886] nix-grpc-daemon[774]: ts=2026-09-22T16:54:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 38.592078] nix-grpc-daemon[767]: ts=2026-09-22T16:54:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.597048] nix-grpc-daemon[767]: ts=2026-09-22T16:54:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.662702] nix-grpc-daemon[774]: ts=2026-09-22T16:54:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.667872] nix-grpc-daemon[774]: ts=2026-09-22T16:54:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 38.797745] nix-grpc-daemon[767]: ts=2026-09-22T16:54:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.802920] nix-grpc-daemon[767]: ts=2026-09-22T16:54:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 38.868855] nix-grpc-daemon[774]: ts=2026-09-22T16:54:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.873754] nix-grpc-daemon[774]: ts=2026-09-22T16:54:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 39.005093] nix-grpc-daemon[767]: ts=2026-09-22T16:54:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.010919] nix-grpc-daemon[767]: ts=2026-09-22T16:54:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 39.076737] nix-grpc-daemon[774]: ts=2026-09-22T16:54:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.080896] nix-grpc-daemon[774]: ts=2026-09-22T16:54:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 39.210594] nix-grpc-daemon[767]: ts=2026-09-22T16:54:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.215921] nix-grpc-daemon[767]: ts=2026-09-22T16:54:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 39.330232] nix-grpc-daemon[767]: ts=2026-09-22T16:54:19Z level=info event=scheduler_take_over worker2 # [ 39.277397] nix-grpc-daemon[774]: ts=2026-09-22T16:54:19Z level=info event=scheduler_connected addr=lb:50051 lb # [ 39.306516] niks3-server[1016]: 2026/09/22 16:54:19 INFO lead: acquired remote=192.168.1.3:52562 worker2 # [ 39.282476] nix-grpc-daemon[774]: ts=2026-09-22T16:54:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 39.416953] nix-grpc-daemon[767]: ts=2026-09-22T16:54:20Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.423836] nix-grpc-daemon[767]: ts=2026-09-22T16:54:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 39.483463] nix-grpc-daemon[774]: ts=2026-09-22T16:54:20Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.560158] nix-grpc-daemon[767]: ts=2026-09-22T16:54:20Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 39.565324] nix-grpc-daemon[767]: ts=2026-09-22T16:54:20Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:45386 worker1 # [ 39.570636] nix-grpc-daemon[767]: ts=2026-09-22T16:54:20Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 39.623840] nix-grpc-daemon[767]: ts=2026-09-22T16:54:20Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.629086] nix-grpc-daemon[767]: ts=2026-09-22T16:54:20Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:45386 worker1 # [ 39.633632] nix-grpc-daemon[767]: ts=2026-09-22T16:54:20Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open' worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.06 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 10.34 seconds) subtest: balancer auth: client cert, bearer token, nothing, unknown CN worker2 # [ 40.938509] nix-grpc-daemon[774]: ts=2026-09-22T16:54:21Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 40.954085] systemd[1]: Started Nix Daemon. worker2 # [ 41.141042] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 41.162496] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 93 ms (attempt 1/5) worker2 # [ 41.259183] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 151 ms (attempt 2/5) worker2 # [ 41.412409] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 31 ms (attempt 3/5) worker2 # [ 41.445101] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 710 ms (attempt 4/5) worker2 # [ 42.159645] nix-grpc-daemon[774]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 42.160005] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 42.183357] nix-grpc-daemon[774]: ts=2026-09-22T16:54:22Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 42.197172] nix-grpc-daemon[774]: ts=2026-09-22T16:54:22Z level=debug event=path_info_miss path=c7dwh1z1cl7zl3zmm2gm23k4b20rxnfn-nixos-system-client-test error="error: path '/nix/store/c7dwh1z1cl7zl3zmm2gm23k4b20rxnfn-nixos-system-client-test' is required, but there is no substituter that can build it" worker2 # [ 42.210890] nix-grpc-daemon[774]: ts=2026-09-22T16:54:22Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 42.218346] nix-grpc-daemon[774]: ts=2026-09-22T16:54:22Z level=debug event=path_info_miss path=c7dwh1z1cl7zl3zmm2gm23k4b20rxnfn-nixos-system-client-test error="error: path '/nix/store/c7dwh1z1cl7zl3zmm2gm23k4b20rxnfn-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.08 seconds) client: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key client: (finished: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds) worker2 # [ 42.544264] nix-grpc-daemon[774]: ts=2026-09-22T16:54:23Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 42.552283] nix-grpc-daemon[774]: ts=2026-09-22T16:54:23Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:40630 worker2 # [ 42.564166] nix-grpc-daemon[774]: ts=2026-09-22T16:54:23Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:40630 worker2 # [ 42.576144] nix-grpc-daemon[774]: ts=2026-09-22T16:54:23Z level=debug event=path_info_miss path=c7dwh1z1cl7zl3zmm2gm23k4b20rxnfn-nixos-system-client-test error="error: path '/nix/store/c7dwh1z1cl7zl3zmm2gm23k4b20rxnfn-nixos-system-client-test' is required, but there is no substituter that can build it" worker1 # [ 42.652672] nix-grpc-daemon[767]: ts=2026-09-22T16:54:23Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:36696 worker1 # [ 42.665160] systemd[1]: Started Nix Daemon. worker1 # [ 42.839778] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 42.856100] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 14 ms (attempt 1/5) worker1 # [ 42.873359] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 126 ms (attempt 2/5) worker1 # [ 43.004444] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 218 ms (attempt 3/5) worker1 # [ 43.231642] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 251 ms (attempt 4/5) worker1 # [ 43.485515] nix-grpc-daemon[767]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 43.490393] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 43.501630] nix-grpc-daemon[767]: ts=2026-09-22T16:54:24Z level=debug event=path_info_miss path=c7dwh1z1cl7zl3zmm2gm23k4b20rxnfn-nixos-system-client-test error="error: path '/nix/store/c7dwh1z1cl7zl3zmm2gm23k4b20rxnfn-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: rm -r /var/lib/nix-grpc-store client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.05 seconds) worker1 # [ 45.814813] nix-grpc-daemon[767]: ts=2026-09-22T16:54:26Z level=info event=denied method=QueryMissing cn=- role=none worker2 # [ 45.987641] nix-grpc-daemon[774]: ts=2026-09-22T16:54:26Z level=info event=denied method=QueryMissing cn=stranger role=none (finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 5.34 seconds) subtest: DAG build is scheduled across workers and lands in the cache client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 worker1 # [ 46.256207] nix-grpc-daemon[767]: ts=2026-09-22T16:54:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:36696 client # don't know how to build these paths: client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker1 # [ 46.275271] nix-grpc-daemon[767]: ts=2026-09-22T16:54:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 46.334592] nix-grpc-daemon[767]: ts=2026-09-22T16:54:26Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=0 worker1 # [ 46.341050] nix-grpc-daemon[767]: ts=2026-09-22T16:54:26Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 46.339383] nix-grpc-daemon[774]: ts=2026-09-22T16:54:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 46.367483] nix-grpc-daemon[774]: ts=2026-09-22T16:54:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 client # copying 1 paths to 'lb:50051'... worker2 # [ 46.387006] nix-grpc-daemon[774]: ts=2026-09-22T16:54:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'... worker2 # [ 46.416969] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 46.461752] nix-grpc-daemon[774]: ts=2026-09-22T16:54:27Z level=info event=niks3_push_started pid=857 worker2 # [ 46.664377] nix-grpc-daemon[857]: time=2026-09-22T16:54:27.309Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 46.781837] nix-daemon[843]: accepted connection from pid 870, user nix-grpc-daemon (trusted) worker2 # [ 46.804473] nix-daemon[843]: reaped child process 873, status = succeeded lb # [ 46.839942] niks3-server[1016]: 2026/09/22 16:54:27 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 46.852074] nix-grpc-daemon[857]: time=2026-09-22T16:54:27.498Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 46.854194] nix-grpc-daemon[857]: time=2026-09-22T16:54:27.498Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)" lb # [ 46.948805] niks3-server[1016]: 2026/09/22 16:54:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign lb # [ 46.982139] niks3-server[1016]: 2026/09/22 16:54:27 INFO Signed narinfos id=1 count=1 worker2 # [ 46.956437] nix-grpc-daemon[857]: time=2026-09-22T16:54:27.603Z level=INFO msg="Uploading 1 narinfos" lb # [ 46.989731] niks3-server[1016]: 2026/09/22 16:54:27 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls lb # [ 47.030378] niks3-server[1016]: 2026/09/22 16:54:27 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete lb # [ 47.037039] niks3-server[1016]: 2026/09/22 16:54:27 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst lb # [ 47.048186] niks3-server[1016]: 2026/09/22 16:54:27 INFO Completed upload id=1 worker2 # [ 47.021254] nix-grpc-daemon[857]: time=2026-09-22T16:54:27.668Z level=INFO msg="Upload complete. (357ms)" lb # [ 47.053327] niks3-server[1016]: 2026/09/22 16:54:27 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo worker2 # [ 47.033609] nix-grpc-daemon[774]: ts=2026-09-22T16:54:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=0 paths=1 nar_bytes_in=504 worker2 # [ 47.057086] nix-grpc-daemon[774]: ts=2026-09-22T16:54:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 47.066405] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 47.073525] nix-daemon[843]: reaped child process 855, status = succeeded client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051... client # farm-a> worker2: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker2 # [ 47.090057] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-t1 worker2 # [ 47.995499] nix-daemon[843]: reaped child process 877, status = succeeded worker2 # [ 48.064712] nix-daemon[843]: accepted connection from pid 893, user nix-grpc-daemon (trusted) worker2 # [ 48.080741] nix-daemon[843]: reaped child process 896, status = succeeded lb # [ 48.117835] niks3-server[1016]: 2026/09/22 16:54:28 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 48.104473] nix-grpc-daemon[857]: time=2026-09-22T16:54:28.749Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 48.106212] nix-grpc-daemon[857]: time=2026-09-22T16:54:28.749Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)" lb # [ 48.165532] niks3-server[1016]: 2026/09/22 16:54:28 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst worker2 # [ 48.151756] nix-grpc-daemon[857]: time=2026-09-22T16:54:28.798Z level=INFO msg="Uploading 1 narinfos" lb # [ 48.179347] niks3-server[1016]: 2026/09/22 16:54:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign lb # [ 48.191865] niks3-server[1016]: 2026/09/22 16:54:28 INFO Signed narinfos id=2 count=1 lb # [ 48.197810] niks3-server[1016]: 2026/09/22 16:54:28 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv lb # [ 48.208892] niks3-server[1016]: 2026/09/22 16:54:28 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete lb # [ 48.215228] niks3-server[1016]: 2026/09/22 16:54:28 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls lb # [ 48.219205] niks3-server[1016]: 2026/09/22 16:54:28 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo lb # [ 48.223698] niks3-server[1016]: 2026/09/22 16:54:28 INFO Completed upload id=2 worker2 # [ 48.195804] nix-grpc-daemon[857]: time=2026-09-22T16:54:28.843Z level=INFO msg="Upload complete. (292ms)" worker2 # [ 48.209164] nix-grpc-daemon[774]: ts=2026-09-22T16:54:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=1 worker1 # [ 48.269436] nix-grpc-daemon[767]: ts=2026-09-22T16:54:28Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=0 worker1 # [ 48.275386] nix-grpc-daemon[767]: ts=2026-09-22T16:54:28Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 48.229245] nix-daemon[843]: reaped child process 875, status = succeeded worker2 # [ 48.261744] nix-grpc-daemon[774]: ts=2026-09-22T16:54:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 48.277075] nix-grpc-daemon[774]: ts=2026-09-22T16:54:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 48.291455] nix-grpc-daemon[857]: time=2026-09-22T16:54:28.938Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 48.302015] nix-grpc-daemon[774]: ts=2026-09-22T16:54:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'... worker2 # [ 48.324326] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 48.483889] nix-daemon[843]: accepted connection from pid 900, user nix-grpc-daemon (trusted) worker2 # [ 48.501333] nix-daemon[843]: reaped child process 903, status = succeeded lb # [ 48.534538] niks3-server[1016]: 2026/09/22 16:54:29 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 48.516456] nix-grpc-daemon[857]: time=2026-09-22T16:54:29.163Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 48.517934] nix-grpc-daemon[857]: time=2026-09-22T16:54:29.163Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)" lb # [ 48.570763] niks3-server[1016]: 2026/09/22 16:54:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign worker2 # [ 48.554778] nix-grpc-daemon[857]: time=2026-09-22T16:54:29.201Z level=INFO msg="Uploading 1 narinfos" lb # [ 48.584572] niks3-server[1016]: 2026/09/22 16:54:29 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls lb # [ 48.593328] niks3-server[1016]: 2026/09/22 16:54:29 INFO Signed narinfos id=3 count=1 lb # [ 48.595775] niks3-server[1016]: 2026/09/22 16:54:29 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst lb # [ 48.610253] niks3-server[1016]: 2026/09/22 16:54:29 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete worker2 # [ 48.582876] nix-grpc-daemon[857]: time=2026-09-22T16:54:29.228Z level=INFO msg="Upload complete. (240ms)" lb # [ 48.614174] niks3-server[1016]: 2026/09/22 16:54:29 INFO Completed upload id=3 lb # [ 48.617213] niks3-server[1016]: 2026/09/22 16:54:29 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo worker2 # [ 48.592454] nix-grpc-daemon[774]: ts=2026-09-22T16:54:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=0 paths=1 nar_bytes_in=624 worker2 # [ 48.608955] nix-grpc-daemon[774]: ts=2026-09-22T16:54:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 48.618315] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 48.622562] nix-daemon[843]: reaped child process 898, status = succeeded client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051... worker2 # [ 48.634188] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) client # farm-b> worker2: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv lb # [ 49.148181] envoy[895]: conn peer=[2001:db8:1::1]:41108 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=715 tx=365 ms=1307 lb # [ 49.153123] envoy[895]: conn peer=[2001:db8:1::1]:41110 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2630 tx=422 ms=925 lb # [ 49.157689] envoy[895]: conn peer=[2001:db8:1::1]:41126 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=199 ms=22 lb # [ 49.163577] envoy[895]: conn peer=[2001:db8:1::1]:41130 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=24 client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-t1 worker2 # [ 50.517076] nix-daemon[843]: reaped child process 907, status = succeeded worker2 # [ 50.625939] nix-daemon[843]: accepted connection from pid 923, user nix-grpc-daemon (trusted) worker2 # [ 50.642716] nix-daemon[843]: reaped child process 926, status = succeeded lb # [ 50.678893] niks3-server[1016]: 2026/09/22 16:54:31 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 50.665660] nix-grpc-daemon[857]: time=2026-09-22T16:54:31.311Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 50.667524] nix-grpc-daemon[857]: time=2026-09-22T16:54:31.311Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)" worker2 # [ 50.709105] nix-grpc-daemon[857]: time=2026-09-22T16:54:31.356Z level=INFO msg="Uploading 1 narinfos" lb # [ 50.727553] niks3-server[1016]: 2026/09/22 16:54:31 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls lb # [ 50.740612] niks3-server[1016]: 2026/09/22 16:54:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign lb # [ 50.752326] niks3-server[1016]: 2026/09/22 16:54:31 INFO Signed narinfos id=4 count=1 lb # [ 50.755023] niks3-server[1016]: 2026/09/22 16:54:31 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst lb # [ 50.763838] niks3-server[1016]: 2026/09/22 16:54:31 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker2 # [ 50.745502] nix-grpc-daemon[857]: time=2026-09-22T16:54:31.392Z level=INFO msg="Upload complete. (294ms)" lb # [ 50.773905] niks3-server[1016]: 2026/09/22 16:54:31 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete lb # [ 50.778017] niks3-server[1016]: 2026/09/22 16:54:31 INFO Completed upload id=4 lb # [ 50.781485] niks3-server[1016]: 2026/09/22 16:54:31 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo worker2 # [ 50.757166] nix-grpc-daemon[774]: ts=2026-09-22T16:54:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=2 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=1 worker1 # [ 50.818829] nix-grpc-daemon[767]: ts=2026-09-22T16:54:31Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=0 worker1 # [ 50.824616] nix-grpc-daemon[767]: ts=2026-09-22T16:54:31Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 50.777591] nix-daemon[843]: reaped child process 905, status = succeeded worker2 # [ 50.812880] nix-grpc-daemon[774]: ts=2026-09-22T16:54:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 50.829046] nix-grpc-daemon[774]: ts=2026-09-22T16:54:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 50.844527] nix-grpc-daemon[857]: time=2026-09-22T16:54:31.491Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'... worker2 # [ 50.855114] nix-grpc-daemon[774]: ts=2026-09-22T16:54:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 50.868882] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 51.024225] nix-daemon[843]: accepted connection from pid 930, user nix-grpc-daemon (trusted) worker2 # [ 51.044721] nix-daemon[843]: reaped child process 933, status = succeeded lb # [ 51.077848] niks3-server[1016]: 2026/09/22 16:54:31 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 51.059831] nix-grpc-daemon[857]: time=2026-09-22T16:54:31.706Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 51.061753] nix-grpc-daemon[857]: time=2026-09-22T16:54:31.706Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)" lb # [ 51.114243] niks3-server[1016]: 2026/09/22 16:54:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign worker2 # [ 51.093362] nix-grpc-daemon[857]: time=2026-09-22T16:54:31.740Z level=INFO msg="Uploading 1 narinfos" lb # [ 51.128368] niks3-server[1016]: 2026/09/22 16:54:31 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls lb # [ 51.145953] niks3-server[1016]: 2026/09/22 16:54:31 INFO Signed narinfos id=5 count=1 lb # [ 51.152704] niks3-server[1016]: 2026/09/22 16:54:31 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst worker2 # [ 51.133460] nix-grpc-daemon[857]: time=2026-09-22T16:54:31.780Z level=INFO msg="Upload complete. (247ms)" lb # [ 51.162244] niks3-server[1016]: 2026/09/22 16:54:31 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete lb # [ 51.169264] niks3-server[1016]: 2026/09/22 16:54:31 INFO Completed upload id=5 worker2 # [ 51.143260] nix-grpc-daemon[774]: ts=2026-09-22T16:54:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 51.173817] niks3-server[1016]: 2026/09/22 16:54:31 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo worker2 # [ 51.156615] nix-grpc-daemon[774]: ts=2026-09-22T16:54:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 51.170179] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 51.175433] nix-daemon[843]: reaped child process 928, status = succeeded client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051... client # farm-top> worker2: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker2 # [ 51.189998] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1 worker2 # [ 52.660104] nix-daemon[843]: reaped child process 937, status = succeeded worker2 # [ 52.770585] nix-daemon[843]: accepted connection from pid 953, user nix-grpc-daemon (trusted) worker2 # [ 52.792051] nix-daemon[843]: reaped child process 956, status = succeeded lb # [ 52.818737] niks3-server[1016]: 2026/09/22 16:54:33 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 52.803191] nix-grpc-daemon[857]: time=2026-09-22T16:54:33.448Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 52.804895] nix-grpc-daemon[857]: time=2026-09-22T16:54:33.448Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)" lb # [ 52.856231] niks3-server[1016]: 2026/09/22 16:54:33 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv lb # [ 52.876253] niks3-server[1016]: 2026/09/22 16:54:33 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls worker2 # [ 52.857396] nix-grpc-daemon[857]: time=2026-09-22T16:54:33.504Z level=INFO msg="Uploading 1 narinfos" lb # [ 52.885652] niks3-server[1016]: 2026/09/22 16:54:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign lb # [ 52.893336] niks3-server[1016]: 2026/09/22 16:54:33 INFO Signed narinfos id=6 count=1 lb # [ 52.898306] niks3-server[1016]: 2026/09/22 16:54:33 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst lb # [ 52.909294] niks3-server[1016]: 2026/09/22 16:54:33 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete lb # [ 52.916852] niks3-server[1016]: 2026/09/22 16:54:33 INFO Completed upload id=6 worker2 # [ 52.887226] nix-grpc-daemon[857]: time=2026-09-22T16:54:33.534Z level=INFO msg="Upload complete. (285ms)" lb # [ 52.918653] niks3-server[1016]: 2026/09/22 16:54:33 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo worker2 # [ 52.901230] nix-grpc-daemon[774]: ts=2026-09-22T16:54:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=1 worker2 # [ 52.919103] nix-daemon[843]: reaped child process 935, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 6.94 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.19 seconds) client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.03 seconds) client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client # copying 3 paths... client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'... client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'... client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.23 seconds) (finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.49 seconds) subtest: repeat is answered Cached by the scheduler without building worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker1 # finding garbage collector roots... worker1 # deleting specified paths... worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker1 # deleting unused links... worker1 # note: hard linking is currently saving 0.0 KiB worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.24 seconds) worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker2 # finding garbage collector roots... worker2 # deleting specified paths... worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker2 # deleting unused links... worker2 # note: hard linking is currently saving 0.0 KiB worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.22 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 worker2 # [ 54.179764] nix-grpc-daemon[774]: ts=2026-09-22T16:54:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:40630 client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked): client # /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker2 # [ 54.199795] nix-grpc-daemon[774]: ts=2026-09-22T16:54:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 54.210943] nix-grpc-daemon[857]: time=2026-09-22T16:54:34.857Z level=INFO msg="All 2 paths already cached" worker1 # [ 54.282395] nix-grpc-daemon[767]: ts=2026-09-22T16:54:34Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=1 client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 0.30 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.16 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: repeat is answered Cached by the scheduler without building, in 1.20 seconds) subtest: inputs already in the cache are not sent to the scheduler lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2 worker1 # [ 55.138860] nix-grpc-daemon[767]: ts=2026-09-22T16:54:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:36696 client # don't know how to build these paths: client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker1 # [ 55.149712] nix-grpc-daemon[767]: ts=2026-09-22T16:54:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 55.175318] nix-grpc-daemon[767]: ts=2026-09-22T16:54:35Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=0 worker1 # [ 55.181074] nix-grpc-daemon[767]: ts=2026-09-22T16:54:35Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 55.171310] nix-grpc-daemon[774]: ts=2026-09-22T16:54:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 55.187668] nix-grpc-daemon[774]: ts=2026-09-22T16:54:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 55.202421] nix-grpc-daemon[857]: time=2026-09-22T16:54:35.849Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'... worker2 # [ 55.213170] nix-grpc-daemon[774]: ts=2026-09-22T16:54:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 55.228145] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 55.392150] nix-daemon[843]: accepted connection from pid 973, user nix-grpc-daemon (trusted) worker2 # [ 55.410105] nix-daemon[843]: reaped child process 976, status = succeeded lb # [ 55.443630] niks3-server[1016]: 2026/09/22 16:54:36 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 55.425891] nix-grpc-daemon[857]: time=2026-09-22T16:54:36.072Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 55.427452] nix-grpc-daemon[857]: time=2026-09-22T16:54:36.072Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)" lb # [ 55.475224] niks3-server[1016]: 2026/09/22 16:54:36 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst worker2 # [ 55.459523] nix-grpc-daemon[857]: time=2026-09-22T16:54:36.106Z level=INFO msg="Uploading 1 narinfos" lb # [ 55.490507] niks3-server[1016]: 2026/09/22 16:54:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign lb # [ 55.500961] niks3-server[1016]: 2026/09/22 16:54:36 INFO Signed narinfos id=7 count=1 lb # [ 55.503952] niks3-server[1016]: 2026/09/22 16:54:36 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls worker2 # [ 55.490955] nix-grpc-daemon[857]: time=2026-09-22T16:54:36.138Z level=INFO msg="Upload complete. (245ms)" lb # [ 55.518601] niks3-server[1016]: 2026/09/22 16:54:36 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete lb # [ 55.527949] niks3-server[1016]: 2026/09/22 16:54:36 INFO Completed upload id=7 worker2 # [ 55.500725] nix-grpc-daemon[774]: ts=2026-09-22T16:54:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 55.534435] niks3-server[1016]: 2026/09/22 16:54:36 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo worker2 # [ 55.516935] nix-grpc-daemon[774]: ts=2026-09-22T16:54:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 55.526515] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 55.533678] nix-daemon[843]: reaped child process 971, status = succeeded client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051... worker2 # [ 55.547498] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) client # farm-top> worker2: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1b worker2 # [ 56.811373] nix-daemon[843]: reaped child process 980, status = succeeded worker2 # [ 56.934529] nix-daemon[843]: accepted connection from pid 996, user nix-grpc-daemon (trusted) worker2 # [ 56.952989] nix-daemon[843]: reaped child process 999, status = succeeded lb # [ 56.987413] niks3-server[1016]: 2026/09/22 16:54:37 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 56.972593] nix-grpc-daemon[857]: time=2026-09-22T16:54:37.616Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 56.975222] nix-grpc-daemon[857]: time=2026-09-22T16:54:37.616Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)" lb # [ 57.032911] niks3-server[1016]: 2026/09/22 16:54:37 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst worker2 # [ 57.017930] nix-grpc-daemon[857]: time=2026-09-22T16:54:37.664Z level=INFO msg="Uploading 1 narinfos" lb # [ 57.050198] niks3-server[1016]: 2026/09/22 16:54:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign lb # [ 57.058469] niks3-server[1016]: 2026/09/22 16:54:37 INFO Signed narinfos id=8 count=1 lb # [ 57.063452] niks3-server[1016]: 2026/09/22 16:54:37 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls lb # [ 57.076508] niks3-server[1016]: 2026/09/22 16:54:37 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv lb # [ 57.079472] niks3-server[1016]: 2026/09/22 16:54:37 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete lb # [ 57.084861] niks3-server[1016]: 2026/09/22 16:54:37 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo lb # [ 57.091105] niks3-server[1016]: 2026/09/22 16:54:37 INFO Completed upload id=8 worker2 # [ 57.063947] nix-grpc-daemon[857]: time=2026-09-22T16:54:37.711Z level=INFO msg="Upload complete. (302ms)" worker2 # [ 57.077066] nix-grpc-daemon[774]: ts=2026-09-22T16:54:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=1 worker2 # [ 57.095466] nix-daemon[843]: reaped child process 978, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.24 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.18 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds) (finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.78 seconds) subtest: outputs read back through any worker carry the cache signature client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2 client # don't know how to build these paths: client # /nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker1 # [ 57.747444] nix-grpc-daemon[767]: ts=2026-09-22T16:54:38Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:36696 worker2 # [ 57.698054] nix-grpc-daemon[774]: ts=2026-09-22T16:54:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 worker1 # [ 57.777483] nix-grpc-daemon[767]: ts=2026-09-22T16:54:38Z level=debug event=want client=4 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv cached=0 worker1 # [ 57.785009] nix-grpc-daemon[767]: ts=2026-09-22T16:54:38Z level=debug event=assigned drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 57.794632] nix-grpc-daemon[767]: ts=2026-09-22T16:54:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 57.806142] nix-grpc-daemon[767]: ts=2026-09-22T16:54:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:36696 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' to 'lb:50051'... worker1 # [ 57.821816] nix-grpc-daemon[767]: ts=2026-09-22T16:54:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 57.837640] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 57.865431] nix-grpc-daemon[767]: ts=2026-09-22T16:54:38Z level=info event=niks3_push_started pid=902 worker1 # [ 58.034767] nix-grpc-daemon[902]: time=2026-09-22T16:54:38.669Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 58.143724] nix-daemon[851]: accepted connection from pid 915, user nix-grpc-daemon (trusted) worker1 # [ 58.166446] nix-daemon[851]: reaped child process 918, status = succeeded lb # [ 58.146664] niks3-server[1016]: 2026/09/22 16:54:38 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 58.183684] nix-grpc-daemon[902]: time=2026-09-22T16:54:38.817Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 58.187701] nix-grpc-daemon[902]: time=2026-09-22T16:54:38.817Z level=INFO msg="Uploading 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv (512B)" lb # [ 58.188000] niks3-server[1016]: 2026/09/22 16:54:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign worker1 # [ 58.223579] nix-grpc-daemon[902]: time=2026-09-22T16:54:38.858Z level=INFO msg="Uploading 1 narinfos" lb # [ 58.199397] niks3-server[1016]: 2026/09/22 16:54:38 INFO Signed narinfos id=9 count=1 lb # [ 58.208485] niks3-server[1016]: 2026/09/22 16:54:38 INFO Registered completed upload object_key=nar/13mhmmy2qay13imbbdi287vhryz1hwnyg0w6agd9s0ph37wnsqll.nar.zst lb # [ 58.221863] niks3-server[1016]: 2026/09/22 16:54:38 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete worker1 # [ 58.257364] nix-grpc-daemon[902]: time=2026-09-22T16:54:38.892Z level=INFO msg="Upload complete. (218ms)" lb # [ 58.234257] niks3-server[1016]: 2026/09/22 16:54:38 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.ls lb # [ 58.242589] niks3-server[1016]: 2026/09/22 16:54:38 INFO Completed upload id=9 worker1 # [ 58.268079] nix-grpc-daemon[767]: ts=2026-09-22T16:54:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 58.245194] niks3-server[1016]: 2026/09/22 16:54:38 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.narinfo worker1 # [ 58.281050] nix-grpc-daemon[767]: ts=2026-09-22T16:54:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 58.287261] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 58.291994] nix-daemon[851]: reaped child process 900, status = succeeded client # building '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' on lb:50051... worker1 # [ 58.303458] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) client # farm-a> node-a: building 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv lb # [ 59.146420] envoy[895]: conn peer=[2001:db8:1::1]:41144 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9575 tx=4392 ms=6722 lb # [ 59.151257] envoy[895]: conn peer=[2001:db8:1::1]:44388 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=480 ms=74 lb # [ 59.156073] envoy[895]: conn peer=[2001:db8:1::1]:44392 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4185 tx=1897 ms=2001 client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-sig worker1 # [ 59.792663] nix-daemon[851]: reaped child process 922, status = succeeded worker1 # [ 59.893378] nix-daemon[851]: accepted connection from pid 938, user nix-grpc-daemon (trusted) worker1 # [ 59.909380] nix-daemon[851]: reaped child process 941, status = succeeded lb # [ 59.894260] niks3-server[1016]: 2026/09/22 16:54:40 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 59.927833] nix-grpc-daemon[902]: time=2026-09-22T16:54:40.560Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 59.931019] nix-grpc-daemon[902]: time=2026-09-22T16:54:40.560Z level=INFO msg="Uploading 2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a (128B)" lb # [ 59.928369] niks3-server[1016]: 2026/09/22 16:54:40 INFO Registered completed upload object_key=log/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv lb # [ 59.939842] niks3-server[1016]: 2026/09/22 16:54:40 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst worker1 # [ 59.975351] nix-grpc-daemon[902]: time=2026-09-22T16:54:40.610Z level=INFO msg="Uploading 1 narinfos" lb # [ 59.953276] niks3-server[1016]: 2026/09/22 16:54:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign lb # [ 59.962392] niks3-server[1016]: 2026/09/22 16:54:40 INFO Signed narinfos id=10 count=1 lb # [ 59.971241] niks3-server[1016]: 2026/09/22 16:54:40 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.ls lb # [ 59.979536] niks3-server[1016]: 2026/09/22 16:54:40 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete lb # [ 59.984077] niks3-server[1016]: 2026/09/22 16:54:40 INFO Completed upload id=10 lb # [ 59.987305] niks3-server[1016]: 2026/09/22 16:54:40 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.narinfo worker1 # [ 60.003563] nix-grpc-daemon[902]: time=2026-09-22T16:54:40.638Z level=INFO msg="Upload complete. (285ms)" worker1 # [ 60.021577] nix-grpc-daemon[767]: ts=2026-09-22T16:54:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=1 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv assign_id=5 outputs=1 worker1 # [ 60.042009] nix-grpc-daemon[767]: ts=2026-09-22T16:54:40Z level=debug event=want client=4 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv cached=0 worker1 # [ 60.046807] nix-grpc-daemon[767]: ts=2026-09-22T16:54:40Z level=debug event=assigned drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 60.051793] nix-daemon[851]: reaped child process 920, status = succeeded worker1 # [ 60.081404] nix-grpc-daemon[767]: ts=2026-09-22T16:54:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 60.096781] nix-grpc-daemon[767]: ts=2026-09-22T16:54:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 60.110753] nix-grpc-daemon[902]: time=2026-09-22T16:54:40.746Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' to 'lb:50051'... worker1 # [ 60.121477] nix-grpc-daemon[767]: ts=2026-09-22T16:54:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 60.140465] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 60.284831] nix-daemon[851]: accepted connection from pid 947, user nix-grpc-daemon (trusted) worker1 # [ 60.306150] nix-daemon[851]: reaped child process 950, status = succeeded lb # [ 60.284946] niks3-server[1016]: 2026/09/22 16:54:40 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 60.317592] nix-grpc-daemon[902]: time=2026-09-22T16:54:40.952Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 60.319198] nix-grpc-daemon[902]: time=2026-09-22T16:54:40.952Z level=INFO msg="Uploading z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv (624B)" lb # [ 60.317896] niks3-server[1016]: 2026/09/22 16:54:40 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.ls lb # [ 60.330401] niks3-server[1016]: 2026/09/22 16:54:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign worker1 # [ 60.361845] nix-grpc-daemon[902]: time=2026-09-22T16:54:40.996Z level=INFO msg="Uploading 1 narinfos" lb # [ 60.342805] niks3-server[1016]: 2026/09/22 16:54:40 INFO Signed narinfos id=11 count=1 lb # [ 60.348384] niks3-server[1016]: 2026/09/22 16:54:40 INFO Registered completed upload object_key=nar/0cnhkgsbmxv5hj7zw3fa59qfxikw6251mlhz2gz1v6pgn8c8shwp.nar.zst worker1 # [ 60.382182] nix-grpc-daemon[902]: time=2026-09-22T16:54:41.017Z level=INFO msg="Upload complete. (231ms)" lb # [ 60.359546] niks3-server[1016]: 2026/09/22 16:54:41 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete lb # [ 60.364626] niks3-server[1016]: 2026/09/22 16:54:41 INFO Completed upload id=11 lb # [ 60.367983] niks3-server[1016]: 2026/09/22 16:54:41 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.narinfo worker1 # [ 60.393365] nix-grpc-daemon[767]: ts=2026-09-22T16:54:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 60.407870] nix-grpc-daemon[767]: ts=2026-09-22T16:54:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 60.414541] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 60.419846] nix-daemon[851]: reaped child process 945, status = succeeded client # building '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' on lb:50051... client # farm-b> node-a: building z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker1 # [ 60.435235] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-sig worker1 # [ 62.144316] nix-daemon[851]: reaped child process 954, status = succeeded worker1 # [ 62.216793] nix-daemon[851]: accepted connection from pid 970, user nix-grpc-daemon (trusted) worker1 # [ 62.237372] nix-daemon[851]: reaped child process 973, status = succeeded lb # [ 62.217365] niks3-server[1016]: 2026/09/22 16:54:42 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 62.251675] nix-grpc-daemon[902]: time=2026-09-22T16:54:42.885Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 62.253646] nix-grpc-daemon[902]: time=2026-09-22T16:54:42.885Z level=INFO msg="Uploading h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b (176B)" lb # [ 62.254930] niks3-server[1016]: 2026/09/22 16:54:42 INFO Registered completed upload object_key=nar/0f75gpfkjj0g9xi2113yvvpdw0l16spc2f1xik1ifnzfbrvx4pfk.nar.zst worker1 # [ 62.294740] nix-grpc-daemon[902]: time=2026-09-22T16:54:42.929Z level=INFO msg="Uploading 1 narinfos" lb # [ 62.271670] niks3-server[1016]: 2026/09/22 16:54:42 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.ls lb # [ 62.281675] niks3-server[1016]: 2026/09/22 16:54:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign lb # [ 62.290623] niks3-server[1016]: 2026/09/22 16:54:42 INFO Signed narinfos id=12 count=1 worker1 # [ 62.318992] nix-grpc-daemon[902]: time=2026-09-22T16:54:42.954Z level=INFO msg="Upload complete. (271ms)" lb # [ 62.295506] niks3-server[1016]: 2026/09/22 16:54:42 INFO Registered completed upload object_key=log/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv lb # [ 62.302347] niks3-server[1016]: 2026/09/22 16:54:42 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete lb # [ 62.306264] niks3-server[1016]: 2026/09/22 16:54:42 INFO Completed upload id=12 lb # [ 62.308941] niks3-server[1016]: 2026/09/22 16:54:42 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.narinfo worker1 # [ 62.334315] nix-grpc-daemon[767]: ts=2026-09-22T16:54:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=1 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv assign_id=6 outputs=1 worker1 # [ 62.355233] nix-grpc-daemon[767]: ts=2026-09-22T16:54:42Z level=debug event=want client=4 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv cached=0 worker1 # [ 62.360377] nix-grpc-daemon[767]: ts=2026-09-22T16:54:42Z level=debug event=assigned drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 62.364947] nix-daemon[851]: reaped child process 952, status = succeeded worker1 # [ 62.393682] nix-grpc-daemon[767]: ts=2026-09-22T16:54:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 62.409227] nix-grpc-daemon[767]: ts=2026-09-22T16:54:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 62.424402] nix-grpc-daemon[902]: time=2026-09-22T16:54:43.059Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' to 'lb:50051'... worker1 # [ 62.435849] nix-grpc-daemon[767]: ts=2026-09-22T16:54:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 62.453213] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 62.608114] nix-daemon[851]: accepted connection from pid 979, user nix-grpc-daemon (trusted) worker1 # [ 62.627800] nix-daemon[851]: reaped child process 982, status = succeeded lb # [ 62.601731] niks3-server[1016]: 2026/09/22 16:54:43 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 62.636162] nix-grpc-daemon[902]: time=2026-09-22T16:54:43.271Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 62.637597] nix-grpc-daemon[902]: time=2026-09-22T16:54:43.271Z level=INFO msg="Uploading knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv (752B)" lb # [ 62.635743] niks3-server[1016]: 2026/09/22 16:54:43 INFO Registered completed upload object_key=nar/0lpagyrngav67x80d1hq0njjlkmqdkvqawraz8dsp36ag50v6iwf.nar.zst worker1 # [ 62.668801] nix-grpc-daemon[902]: time=2026-09-22T16:54:43.303Z level=INFO msg="Uploading 1 narinfos" lb # [ 62.651360] niks3-server[1016]: 2026/09/22 16:54:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign lb # [ 62.660441] niks3-server[1016]: 2026/09/22 16:54:43 INFO Signed narinfos id=13 count=1 lb # [ 62.662884] niks3-server[1016]: 2026/09/22 16:54:43 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.ls worker1 # [ 62.693187] nix-grpc-daemon[902]: time=2026-09-22T16:54:43.328Z level=INFO msg="Upload complete. (222ms)" lb # [ 62.674851] niks3-server[1016]: 2026/09/22 16:54:43 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete lb # [ 62.679671] niks3-server[1016]: 2026/09/22 16:54:43 INFO Completed upload id=13 worker1 # [ 62.704870] nix-grpc-daemon[767]: ts=2026-09-22T16:54:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 62.682258] niks3-server[1016]: 2026/09/22 16:54:43 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.narinfo worker1 # [ 62.715657] nix-grpc-daemon[767]: ts=2026-09-22T16:54:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 62.722464] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 62.727099] nix-daemon[851]: reaped child process 977, status = succeeded client # building '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' on lb:50051... client # farm-top> node-a: building knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker1 # [ 62.740853] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-sig worker1 # [ 64.424531] nix-daemon[851]: reaped child process 986, status = succeeded worker1 # [ 64.502775] nix-daemon[851]: accepted connection from pid 1002, user nix-grpc-daemon (trusted) worker1 # [ 64.522517] nix-daemon[851]: reaped child process 1005, status = succeeded lb # [ 64.503265] niks3-server[1016]: 2026/09/22 16:54:45 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 64.538366] nix-grpc-daemon[902]: time=2026-09-22T16:54:45.171Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 64.539967] nix-grpc-daemon[902]: time=2026-09-22T16:54:45.171Z level=INFO msg="Uploading 4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top (232B)" lb # [ 64.546044] niks3-server[1016]: 2026/09/22 16:54:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign worker1 # [ 64.581683] nix-grpc-daemon[902]: time=2026-09-22T16:54:45.216Z level=INFO msg="Uploading 1 narinfos" lb # [ 64.562227] niks3-server[1016]: 2026/09/22 16:54:45 INFO Signed narinfos id=14 count=1 lb # [ 64.573203] niks3-server[1016]: 2026/09/22 16:54:45 INFO Registered completed upload object_key=nar/07ar364rwqqniwi2nkdpqlgv7ws7702n5lqz703ackfmssrh40v8.nar.zst lb # [ 64.584406] niks3-server[1016]: 2026/09/22 16:54:45 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.ls lb # [ 64.595198] niks3-server[1016]: 2026/09/22 16:54:45 INFO Registered completed upload object_key=log/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv lb # [ 64.604409] niks3-server[1016]: 2026/09/22 16:54:45 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete lb # [ 64.610055] niks3-server[1016]: 2026/09/22 16:54:45 INFO Completed upload id=14 worker1 # [ 64.634528] nix-grpc-daemon[902]: time=2026-09-22T16:54:45.269Z level=INFO msg="Upload complete. (282ms)" lb # [ 64.613639] niks3-server[1016]: 2026/09/22 16:54:45 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.narinfo worker1 # [ 64.649149] nix-grpc-daemon[767]: ts=2026-09-22T16:54:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=1 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv assign_id=7 outputs=1 worker1 # [ 64.666554] nix-daemon[851]: reaped child process 984, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2, in 7.16 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath, in 0.18 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker1 # [ 65.001383] nix-grpc-daemon[767]: ts=2026-09-22T16:54:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:45438 worker1 # [ 65.042788] nix-grpc-daemon[767]: ts=2026-09-22T16:54:45Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:45438 client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.24 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker2 # [ 65.243996] nix-grpc-daemon[774]: ts=2026-09-22T16:54:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:37458 client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked): client # /nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a client # /nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b client # /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker2 # [ 65.270389] nix-grpc-daemon[774]: ts=2026-09-22T16:54:45Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:37458 worker2 # [ 65.279672] nix-grpc-daemon[774]: copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'http://lb:5751'... worker2 # [ 65.297174] nix-grpc-daemon[774]: copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'http://lb:5751'... worker2 # [ 65.311277] nix-grpc-daemon[774]: copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.29 seconds) client: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top client # finding garbage collector roots... client # removing stale temporary roots file "/nix/var/nix/temproots/870" client # removing stale temporary roots file "/nix/var/nix/temproots/891" client # removing stale temporary roots file "/nix/var/nix/temproots/932" client # removing stale temporary roots file "/nix/var/nix/temproots/911" client # removing stale temporary roots file "/nix/var/nix/temproots/903" client # removing stale temporary roots file "/nix/var/nix/temproots/947" client # removing stale temporary roots file "/nix/var/nix/temproots/924" client # removing stale temporary roots file "/nix/var/nix/temproots/855" client # deleting specified paths... client # deleting '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' client # deleting unused links... client # note: hard linking is currently saving 0.0 KiB worker1 # [ 65.746551] nix-grpc-daemon[767]: ts=2026-09-22T16:54:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 65.755343] nix-grpc-daemon[767]: ts=2026-09-22T16:54:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 65.766028] nix-grpc-daemon[902]: time=2026-09-22T16:54:46.400Z level=INFO msg="All 1 paths already cached" worker1 # [ 65.774196] nix-grpc-daemon[767]: ts=2026-09-22T16:54:46Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:36696 worker2 # [ 65.731340] nix-grpc-daemon[774]: ts=2026-09-22T16:54:46Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:40630 client # copying 3 paths... worker2 # [ 65.746388] nix-grpc-daemon[774]: ts=2026-09-22T16:54:46Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:40630 client # copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'grpc://lb:50051'... worker1 # [ 65.825371] nix-grpc-daemon[767]: ts=2026-09-22T16:54:46Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 65.832905] nix-grpc-daemon[767]: ts=2026-09-22T16:54:46Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=0 paths=1 nar_bytes_out=232 worker2 # [ 65.788496] nix-grpc-daemon[774]: ts=2026-09-22T16:54:46Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 65.795760] nix-grpc-daemon[774]: ts=2026-09-22T16:54:46Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=0 paths=1 nar_bytes_out=176 worker1 # [ 65.854843] nix-grpc-daemon[767]: ts=2026-09-22T16:54:46Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 65.861693] nix-grpc-daemon[767]: ts=2026-09-22T16:54:46Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=0 paths=1 nar_bytes_out=128 client # copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'grpc://lb:50051'... client # copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'grpc://lb:50051'... client: (finished: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.55 seconds) (finished: subtest: outputs read back through any worker carry the cache signature, in 8.43 seconds) subtest: two clients wanting the same drv build it once client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup client # [ 65.943902] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup1.service; invocation ID: 8434597c6e8244c18f6fd6eeb0ca4d20 client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.07 seconds) client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup worker2 # [ 66.145903] nix-grpc-daemon[774]: ts=2026-09-22T16:54:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:40630 client # [ 66.140848] nix[1000]: don't know how to build these paths: client # [ 66.142569] nix[1000]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker1 # [ 66.215560] nix-grpc-daemon[767]: ts=2026-09-22T16:54:46Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker1 # [ 66.221635] nix-grpc-daemon[767]: ts=2026-09-22T16:54:46Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 66.209177] nix-grpc-daemon[774]: ts=2026-09-22T16:54:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 66.223977] nix-grpc-daemon[774]: ts=2026-09-22T16:54:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 client # [ 66.224230] nix[1000]: copying 1 paths to 'lb:50051'... client # [ 66.229524] nix[1000]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'... worker2 # [ 66.241118] nix-grpc-daemon[774]: ts=2026-09-22T16:54:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 66.257975] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 66.417634] nix-daemon[843]: accepted connection from pid 1006, user nix-grpc-daemon (trusted) worker2 # [ 66.435605] nix-daemon[843]: reaped child process 1009, status = succeeded lb # [ 66.471104] niks3-server[1016]: 2026/09/22 16:54:47 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 66.453243] nix-grpc-daemon[857]: time=2026-09-22T16:54:47.099Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 66.454804] nix-grpc-daemon[857]: time=2026-09-22T16:54:47.099Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)" lb # [ 66.503212] niks3-server[1016]: 2026/09/22 16:54:47 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst worker2 # [ 66.486138] nix-grpc-daemon[857]: time=2026-09-22T16:54:47.133Z level=INFO msg="Uploading 1 narinfos" lb # [ 66.515788] niks3-server[1016]: 2026/09/22 16:54:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign lb # [ 66.527637] niks3-server[1016]: 2026/09/22 16:54:47 INFO Signed narinfos id=15 count=1 lb # [ 66.533486] niks3-server[1016]: 2026/09/22 16:54:47 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls lb # [ 66.544389] niks3-server[1016]: 2026/09/22 16:54:47 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete lb # [ 66.549968] niks3-server[1016]: 2026/09/22 16:54:47 INFO Completed upload id=15 worker2 # [ 66.522435] nix-grpc-daemon[857]: time=2026-09-22T16:54:47.169Z level=INFO msg="Upload complete. (249ms)" lb # [ 66.553160] niks3-server[1016]: 2026/09/22 16:54:47 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo worker2 # [ 66.534570] nix-grpc-daemon[774]: ts=2026-09-22T16:54:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=0 paths=1 nar_bytes_in=432 worker2 # [ 66.553279] nix-grpc-daemon[774]: ts=2026-09-22T16:54:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 66.563480] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 66.572385] nix-daemon[843]: reaped child process 1004, status = succeeded client # [ 66.566334] nix[1000]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... worker2 # [ 66.582398] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) client # [ 67.038290] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup2.service; invocation ID: a76b0024c53b4515956c56ad9a8838c9 client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.11 seconds) client: must succeed: sleep 2 worker1 # [ 67.358653] nix-grpc-daemon[767]: ts=2026-09-22T16:54:47Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:36696 client # [ 67.301986] nix[1016]: don't know how to build these paths: client # [ 67.305209] nix[1016]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker1 # [ 67.381183] nix-grpc-daemon[767]: ts=2026-09-22T16:54:48Z level=debug event=want client=6 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker2 # [ 67.333310] nix-grpc-daemon[774]: ts=2026-09-22T16:54:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 client # [ 67.324508] nix[1016]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... lb # [ 69.147493] envoy[895]: conn peer=[2001:db8:1::1]:44398 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9571 tx=4388 ms=6915 lb # [ 69.152497] envoy[895]: conn peer=[2001:db8:1::1]:49176 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1051 tx=2887 ms=194 lb # [ 69.157254] envoy[895]: conn peer=[2001:db8:1::1]:49216 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=408 tx=329 ms=87 lb # [ 69.161876] envoy[895]: conn peer=[2001:db8:1::1]:49200 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=408 tx=375 ms=102 lb # [ 69.167211] envoy[895]: conn peer=[2001:db8:1::1]:49188 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=384 tx=409 ms=116 client: (finished: must succeed: sleep 2, in 2.07 seconds) client: waiting for success: ! systemctl is-active dedup1 dedup2 worker2 # [ 69.647920] nix-daemon[843]: reaped child process 1013, status = succeeded worker2 # [ 69.775846] nix-daemon[843]: accepted connection from pid 1039, user nix-grpc-daemon (trusted) worker2 # [ 69.797033] nix-daemon[843]: reaped child process 1042, status = succeeded lb # [ 69.823999] niks3-server[1016]: 2026/09/22 16:54:50 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 69.807817] nix-grpc-daemon[857]: time=2026-09-22T16:54:50.453Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 69.809854] nix-grpc-daemon[857]: time=2026-09-22T16:54:50.453Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)" lb # [ 69.864140] niks3-server[1016]: 2026/09/22 16:54:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign worker2 # [ 69.844333] nix-grpc-daemon[857]: time=2026-09-22T16:54:50.491Z level=INFO msg="Uploading 1 narinfos" lb # [ 69.880700] niks3-server[1016]: 2026/09/22 16:54:50 INFO Signed narinfos id=16 count=1 lb # [ 69.884007] niks3-server[1016]: 2026/09/22 16:54:50 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst lb # [ 69.901785] niks3-server[1016]: 2026/09/22 16:54:50 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls lb # [ 69.915401] niks3-server[1016]: 2026/09/22 16:54:50 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker2 # [ 69.887998] nix-grpc-daemon[857]: time=2026-09-22T16:54:50.535Z level=INFO msg="Upload complete. (291ms)" lb # [ 69.921257] niks3-server[1016]: 2026/09/22 16:54:50 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete lb # [ 69.927681] niks3-server[1016]: 2026/09/22 16:54:50 INFO Completed upload id=16 lb # [ 69.931348] niks3-server[1016]: 2026/09/22 16:54:50 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo worker2 # [ 69.901078] nix-grpc-daemon[774]: ts=2026-09-22T16:54:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker2 # [ 69.919921] nix-grpc-daemon[774]: ts=2026-09-22T16:54:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=2 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker2 # [ 69.927458] nix-daemon[843]: reaped child process 1011, status = succeeded client # [ 69.930040] systemd[1]: dedup2.service: Deactivated successfully. client # [ 69.935382] systemd[1]: dedup2.service: Consumed 212ms CPU time over 2.893s wall clock time, 33.2M memory peak, 5.9K incoming IP traffic, 5.6K outgoing IP traffic. client # [ 69.976324] systemd[1]: dedup1.service: Deactivated successfully. client # [ 69.979693] systemd[1]: dedup1.service: Consumed 171ms CPU time over 4.035s wall clock time, 43.5M memory peak, 7.3K incoming IP traffic, 8.4K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.12 seconds) client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; } client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.06 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: two clients wanting the same drv build it once, in 4.98 seconds) subtest: --timeout and --max-silent-time reach the builder worker1 # [ 71.173233] nix-grpc-daemon[767]: ts=2026-09-22T16:54:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 71.189558] nix-grpc-daemon[767]: ts=2026-09-22T16:54:51Z level=debug event=want client=7 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv cached=0 worker1 # [ 71.196762] nix-grpc-daemon[767]: ts=2026-09-22T16:54:51Z level=debug event=assigned drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 71.205323] nix-grpc-daemon[767]: ts=2026-09-22T16:54:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 71.216309] nix-grpc-daemon[767]: ts=2026-09-22T16:54:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 71.230022] nix-grpc-daemon[767]: ts=2026-09-22T16:54:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 71.245872] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 71.398995] nix-daemon[851]: accepted connection from pid 1032, user nix-grpc-daemon (trusted) worker1 # [ 71.419426] nix-daemon[851]: reaped child process 1035, status = succeeded lb # [ 71.399556] niks3-server[1016]: 2026/09/22 16:54:52 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 71.433430] nix-grpc-daemon[902]: time=2026-09-22T16:54:52.068Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 71.434976] nix-grpc-daemon[902]: time=2026-09-22T16:54:52.068Z level=INFO msg="Uploading sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv (424B)" lb # [ 71.432927] niks3-server[1016]: 2026/09/22 16:54:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign worker1 # [ 71.463859] nix-grpc-daemon[902]: time=2026-09-22T16:54:52.099Z level=INFO msg="Uploading 1 narinfos" lb # [ 71.444995] niks3-server[1016]: 2026/09/22 16:54:52 INFO Signed narinfos id=17 count=1 lb # [ 71.453861] niks3-server[1016]: 2026/09/22 16:54:52 INFO Registered completed upload object_key=nar/1m12xmj6yzmb9y2j8ly0yvh5fyzy1nqk7ajhqwr4wn3b5h0bsch5.nar.zst lb # [ 71.464783] niks3-server[1016]: 2026/09/22 16:54:52 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.ls lb # [ 71.479211] niks3-server[1016]: 2026/09/22 16:54:52 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete worker1 # [ 71.509364] nix-grpc-daemon[902]: time=2026-09-22T16:54:52.144Z level=INFO msg="Upload complete. (248ms)" lb # [ 71.486550] niks3-server[1016]: 2026/09/22 16:54:52 INFO Completed upload id=17 lb # [ 71.493853] niks3-server[1016]: 2026/09/22 16:54:52 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.narinfo worker1 # [ 71.520371] nix-grpc-daemon[767]: ts=2026-09-22T16:54:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=0 paths=1 nar_bytes_in=424 worker1 # [ 71.540150] nix-grpc-daemon[767]: ts=2026-09-22T16:54:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 71.548383] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 71.555136] nix-daemon[851]: reaped child process 1030, status = succeeded worker1 # [ 71.569732] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 77.993794] nix-grpc-daemon[767]: ts=2026-09-22T16:54:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=6 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv assign_id=9 outputs=0 worker1 # [ 78.037000] nix-daemon[851]: reaped child process 1037, status = succeeded worker1 # [ 78.039660] nix-daemon[851]: reaped child process 1039, status = succeeded worker1 # [ 78.238497] nix-grpc-daemon[767]: ts=2026-09-22T16:54:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 78.251478] nix-grpc-daemon[767]: ts=2026-09-22T16:54:58Z level=debug event=want client=8 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv cached=0 worker2 # [ 78.205439] nix-grpc-daemon[774]: ts=2026-09-22T16:54:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker1 # [ 78.257260] nix-grpc-daemon[767]: ts=2026-09-22T16:54:58Z level=debug event=assigned drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 78.221270] nix-grpc-daemon[774]: ts=2026-09-22T16:54:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 78.236116] nix-grpc-daemon[774]: ts=2026-09-22T16:54:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 78.251284] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 78.415717] nix-daemon[843]: accepted connection from pid 1055, user nix-grpc-daemon (trusted) worker2 # [ 78.439407] nix-daemon[843]: reaped child process 1058, status = succeeded lb # [ 78.466004] niks3-server[1016]: 2026/09/22 16:54:59 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 78.447809] nix-grpc-daemon[857]: time=2026-09-22T16:54:59.094Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 78.450016] nix-grpc-daemon[857]: time=2026-09-22T16:54:59.096Z level=INFO msg="Uploading pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv (424B)" lb # [ 78.502524] niks3-server[1016]: 2026/09/22 16:54:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign worker2 # [ 78.484082] nix-grpc-daemon[857]: time=2026-09-22T16:54:59.131Z level=INFO msg="Uploading 1 narinfos" lb # [ 78.514052] niks3-server[1016]: 2026/09/22 16:54:59 INFO Signed narinfos id=18 count=1 lb # [ 78.524454] niks3-server[1016]: 2026/09/22 16:54:59 INFO Registered completed upload object_key=nar/14f5yrnllfav3svmw4v4svw15r4zlf5yv1p972jy14mmq6gdw943.nar.zst lb # [ 78.536515] niks3-server[1016]: 2026/09/22 16:54:59 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.ls worker2 # [ 78.521750] nix-grpc-daemon[857]: time=2026-09-22T16:54:59.169Z level=INFO msg="Upload complete. (253ms)" lb # [ 78.549564] niks3-server[1016]: 2026/09/22 16:54:59 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete lb # [ 78.559740] niks3-server[1016]: 2026/09/22 16:54:59 INFO Completed upload id=18 worker2 # [ 78.532766] nix-grpc-daemon[774]: ts=2026-09-22T16:54:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=0 paths=1 nar_bytes_in=424 lb # [ 78.562613] niks3-server[1016]: 2026/09/22 16:54:59 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.narinfo worker2 # [ 78.547986] nix-grpc-daemon[774]: ts=2026-09-22T16:54:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 78.556472] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 78.559298] nix-daemon[843]: reaped child process 1053, status = succeeded worker2 # [ 78.569637] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) lb # [ 79.148398] envoy[895]: conn peer=[2001:db8:1::1]:49236 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1197 ms=2618 lb # [ 79.148727] envoy[895]: conn peer=[2001:db8:1::1]:49226 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3031 tx=1541 ms=3824 lb # [ 79.148846] envoy[895]: conn peer=[2001:db8:1::1]:33462 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3014 tx=952 ms=6849 worker2 # [ 84.987722] nix-grpc-daemon[774]: ts=2026-09-22T16:55:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=6 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv assign_id=10 outputs=0 worker2 # [ 85.032892] nix-daemon[843]: reaped child process 1060, status = succeeded worker2 # [ 85.035751] nix-daemon[843]: reaped child process 1062, status = succeeded (finished: subtest: --timeout and --max-silent-time reach the builder, in 14.33 seconds) subtest: the first client leaving does not fail the second client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share client # [ 85.276849] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share1.service; invocation ID: 25c8a1141ba34e2fa8474454c4a82020 client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 0.09 seconds) client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share worker2 # [ 85.488154] nix-grpc-daemon[774]: ts=2026-09-22T16:55:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:40630 client # [ 85.483005] nix[1088]: don't know how to build these paths: client # [ 85.483974] nix[1088]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker1 # [ 85.562022] nix-grpc-daemon[767]: ts=2026-09-22T16:55:06Z level=debug event=want client=9 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 worker1 # [ 85.569903] nix-grpc-daemon[767]: ts=2026-09-22T16:55:06Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 85.577859] nix-grpc-daemon[767]: ts=2026-09-22T16:55:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 85.589204] nix-grpc-daemon[767]: ts=2026-09-22T16:55:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:36696 client # [ 85.536470] nix[1088]: copying 1 paths to 'lb:50051'... client # [ 85.541146] nix[1088]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'... worker1 # [ 85.605067] nix-grpc-daemon[767]: ts=2026-09-22T16:55:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 85.620253] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 85.796116] nix-daemon[851]: accepted connection from pid 1066, user nix-grpc-daemon (trusted) worker1 # [ 85.814152] nix-daemon[851]: reaped child process 1069, status = succeeded lb # [ 85.796686] niks3-server[1016]: 2026/09/22 16:55:06 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 85.832430] nix-grpc-daemon[902]: time=2026-09-22T16:55:06.466Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 85.833977] nix-grpc-daemon[902]: time=2026-09-22T16:55:06.466Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)" lb # [ 85.838281] niks3-server[1016]: 2026/09/22 16:55:06 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst worker1 # [ 85.876910] nix-grpc-daemon[902]: time=2026-09-22T16:55:06.511Z level=INFO msg="Uploading 1 narinfos" lb # [ 85.852371] niks3-server[1016]: 2026/09/22 16:55:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign lb # [ 85.862698] niks3-server[1016]: 2026/09/22 16:55:06 INFO Signed narinfos id=19 count=1 lb # [ 85.869140] niks3-server[1016]: 2026/09/22 16:55:06 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls lb # [ 85.879454] niks3-server[1016]: 2026/09/22 16:55:06 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete lb # [ 85.891695] niks3-server[1016]: 2026/09/22 16:55:06 INFO Completed upload id=19 worker1 # [ 85.914728] nix-grpc-daemon[902]: time=2026-09-22T16:55:06.550Z level=INFO msg="Upload complete. (274ms)" lb # [ 85.897618] niks3-server[1016]: 2026/09/22 16:55:06 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo worker1 # [ 85.929120] nix-grpc-daemon[767]: ts=2026-09-22T16:55:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=0 paths=1 nar_bytes_in=432 worker1 # [ 85.944460] nix-grpc-daemon[767]: ts=2026-09-22T16:55:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 85.956561] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 85.959971] nix-daemon[851]: reaped child process 1064, status = succeeded client # [ 85.906859] nix[1088]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... worker1 # [ 85.974463] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) client # [ 86.374644] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share2.service; invocation ID: cdb015f6f89c400b9816512bbd6905cb client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 1.11 seconds) worker1 # [ 86.634532] nix-grpc-daemon[767]: ts=2026-09-22T16:55:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:36696 client # [ 86.584227] nix[1104]: this derivation will be built: worker2 # [ 86.600808] nix-grpc-daemon[774]: ts=2026-09-22T16:55:07Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 86.615470] nix-grpc-daemon[774]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' from 'http://lb:5751'... client # [ 86.626403] nix[1104]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker1 # [ 86.702490] nix-grpc-daemon[767]: ts=2026-09-22T16:55:07Z level=debug event=want client=10 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 worker1 # [ 86.710561] nix-grpc-daemon[767]: ts=2026-09-22T16:55:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 client # [ 86.653671] nix[1104]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... client: must succeed: sleep 2; systemctl kill -s INT share1 lb # [ 89.149308] envoy[895]: conn peer=[2001:db8:1::1]:33468 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3028 tx=953 ms=6860 client # [ 89.748959] systemd[1]: share1.service: Sent signal SIGINT to main process 1088 (nix) on client request. client # [ 89.758549] nix[1088]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.09 seconds) client: waiting for success: ! systemctl is-active share1 client # [ 89.773743] nix[1088]: error: interrupted by the user client # [ 89.837339] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE client # [ 89.840156] systemd[1]: share1.service: Failed with result 'exit-code'. client # [ 89.843212] systemd[1]: share1.service: Consumed 181ms CPU time over 4.564s wall clock time, 41.4M memory peak, 6.2K incoming IP traffic, 8.3K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share1, in 1.12 seconds) worker1 # [ 91.199277] nix-daemon[851]: reaped child process 1073, status = succeeded client: waiting for success: ! systemctl is-active share2 worker1 # [ 91.301729] nix-daemon[851]: accepted connection from pid 1109, user nix-grpc-daemon (trusted) worker1 # [ 91.318797] nix-daemon[851]: reaped child process 1112, status = succeeded lb # [ 91.302964] niks3-server[1016]: 2026/09/22 16:55:11 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 91.340401] nix-grpc-daemon[902]: time=2026-09-22T16:55:11.974Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 91.341846] nix-grpc-daemon[902]: time=2026-09-22T16:55:11.974Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)" lb # [ 91.354784] niks3-server[1016]: 2026/09/22 16:55:12 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls worker1 # [ 91.391445] nix-grpc-daemon[902]: time=2026-09-22T16:55:12.026Z level=INFO msg="Uploading 1 narinfos" lb # [ 91.369095] niks3-server[1016]: 2026/09/22 16:55:12 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst lb # [ 91.378969] niks3-server[1016]: 2026/09/22 16:55:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign lb # [ 91.388372] niks3-server[1016]: 2026/09/22 16:55:12 INFO Signed narinfos id=20 count=1 worker1 # [ 91.412763] nix-grpc-daemon[902]: time=2026-09-22T16:55:12.048Z level=INFO msg="Upload complete. (224ms)" lb # [ 91.390411] niks3-server[1016]: 2026/09/22 16:55:12 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker1 # [ 91.424300] nix-grpc-daemon[767]: ts=2026-09-22T16:55:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=4 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 lb # [ 91.401560] niks3-server[1016]: 2026/09/22 16:55:12 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete lb # [ 91.408409] niks3-server[1016]: 2026/09/22 16:55:12 INFO Completed upload id=20 lb # [ 91.411038] niks3-server[1016]: 2026/09/22 16:55:12 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo worker1 # [ 91.440543] nix-grpc-daemon[767]: ts=2026-09-22T16:55:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=5 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 worker1 # [ 91.447789] nix-daemon[851]: reaped child process 1071, status = succeeded client # [ 91.435522] systemd[1]: share2.service: Deactivated successfully. client # [ 91.440120] systemd[1]: share2.service: Consumed 187ms CPU time over 5.062s wall clock time, 33.3M memory peak, 6.5K incoming IP traffic, 5.9K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share2, in 1.12 seconds) client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; } client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.06 seconds) (finished: subtest: the first client leaving does not fail the second, in 7.11 seconds) subtest: build hook: nix-daemon with builders = grpc://lb client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log client # [ 92.462137] systemd[1]: Started Nix Daemon. client # [ 92.583285] nix-daemon[1156]: accepted connection from pid 1153, user root (trusted) worker2 # [ 92.742261] nix-grpc-daemon[774]: ts=2026-09-22T16:55:13Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 92.749280] nix-grpc-daemon[774]: ts=2026-09-22T16:55:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 worker1 # [ 92.813318] nix-grpc-daemon[767]: ts=2026-09-22T16:55:13Z level=debug event=want client=11 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=0 worker1 # [ 92.819600] nix-grpc-daemon[767]: ts=2026-09-22T16:55:13Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 92.770447] nix-grpc-daemon[774]: ts=2026-09-22T16:55:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 92.785663] nix-grpc-daemon[774]: ts=2026-09-22T16:55:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 92.799203] nix-grpc-daemon[774]: ts=2026-09-22T16:55:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 92.814356] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 92.984188] nix-daemon[843]: accepted connection from pid 1102, user nix-grpc-daemon (trusted) worker2 # [ 93.001361] nix-daemon[843]: reaped child process 1105, status = succeeded lb # [ 93.037590] niks3-server[1016]: 2026/09/22 16:55:13 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 93.020284] nix-grpc-daemon[857]: time=2026-09-22T16:55:13.666Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 93.021760] nix-grpc-daemon[857]: time=2026-09-22T16:55:13.666Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)" lb # [ 93.069897] niks3-server[1016]: 2026/09/22 16:55:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign worker2 # [ 93.050249] nix-grpc-daemon[857]: time=2026-09-22T16:55:13.696Z level=INFO msg="Uploading 1 narinfos" lb # [ 93.082732] niks3-server[1016]: 2026/09/22 16:55:13 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls lb # [ 93.092970] niks3-server[1016]: 2026/09/22 16:55:13 INFO Signed narinfos id=21 count=1 lb # [ 93.101070] niks3-server[1016]: 2026/09/22 16:55:13 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst worker2 # [ 93.081671] nix-grpc-daemon[857]: time=2026-09-22T16:55:13.728Z level=INFO msg="Upload complete. (247ms)" lb # [ 93.111467] niks3-server[1016]: 2026/09/22 16:55:13 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete lb # [ 93.117835] niks3-server[1016]: 2026/09/22 16:55:13 INFO Completed upload id=21 worker2 # [ 93.092348] nix-grpc-daemon[774]: ts=2026-09-22T16:55:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 93.121757] niks3-server[1016]: 2026/09/22 16:55:13 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo worker2 # [ 93.103716] nix-grpc-daemon[774]: ts=2026-09-22T16:55:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 93.113587] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 93.116369] nix-daemon[843]: reaped child process 1100, status = succeeded worker2 # [ 93.130561] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 94.590603] nix-daemon[843]: reaped child process 1109, status = succeeded worker2 # [ 94.721437] nix-daemon[843]: accepted connection from pid 1125, user nix-grpc-daemon (trusted) worker2 # [ 94.742580] nix-daemon[843]: reaped child process 1128, status = succeeded lb # [ 94.771160] niks3-server[1016]: 2026/09/22 16:55:15 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 94.756257] nix-grpc-daemon[857]: time=2026-09-22T16:55:15.400Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 94.757952] nix-grpc-daemon[857]: time=2026-09-22T16:55:15.400Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)" lb # [ 94.818491] niks3-server[1016]: 2026/09/22 16:55:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign worker2 # [ 94.799126] nix-grpc-daemon[857]: time=2026-09-22T16:55:15.446Z level=INFO msg="Uploading 1 narinfos" lb # [ 94.830467] niks3-server[1016]: 2026/09/22 16:55:15 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst lb # [ 94.841299] niks3-server[1016]: 2026/09/22 16:55:15 INFO Signed narinfos id=22 count=1 lb # [ 94.843403] niks3-server[1016]: 2026/09/22 16:55:15 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls lb # [ 94.849313] niks3-server[1016]: 2026/09/22 16:55:15 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv lb # [ 94.857634] niks3-server[1016]: 2026/09/22 16:55:15 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete lb # [ 94.861137] niks3-server[1016]: 2026/09/22 16:55:15 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo worker2 # [ 94.833943] nix-grpc-daemon[857]: time=2026-09-22T16:55:15.481Z level=INFO msg="Upload complete. (292ms)" lb # [ 94.864212] niks3-server[1016]: 2026/09/22 16:55:15 INFO Completed upload id=22 worker2 # [ 94.845245] nix-grpc-daemon[774]: ts=2026-09-22T16:55:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=12 outputs=1 worker2 # [ 94.863699] nix-daemon[843]: reaped child process 1107, status = succeeded worker2 # [ 94.888937] nix-grpc-daemon[774]: ts=2026-09-22T16:55:15Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 94.918628] nix-grpc-daemon[774]: ts=2026-09-22T16:55:15Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 94.925563] nix-grpc-daemon[774]: ts=2026-09-22T16:55:15Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=0 paths=1 nar_bytes_out=128 worker1 # [ 95.265212] nix-grpc-daemon[767]: ts=2026-09-22T16:55:15Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:36696 worker2 # [ 95.216615] nix-grpc-daemon[774]: ts=2026-09-22T16:55:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 95.226434] nix-grpc-daemon[857]: time=2026-09-22T16:55:15.873Z level=INFO msg="All 1 paths already cached" worker1 # [ 95.297532] nix-grpc-daemon[767]: ts=2026-09-22T16:55:15Z level=debug event=want client=12 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=0 worker1 # [ 95.303239] nix-grpc-daemon[767]: ts=2026-09-22T16:55:15Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 95.293809] nix-grpc-daemon[774]: ts=2026-09-22T16:55:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 95.312746] nix-grpc-daemon[774]: ts=2026-09-22T16:55:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 95.330264] nix-grpc-daemon[857]: time=2026-09-22T16:55:15.977Z level=INFO msg="All 1 paths already cached" worker2 # [ 95.342402] nix-grpc-daemon[774]: ts=2026-09-22T16:55:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 95.356235] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 95.520881] nix-daemon[843]: accepted connection from pid 1132, user nix-grpc-daemon (trusted) worker2 # [ 95.541088] nix-daemon[843]: reaped child process 1135, status = succeeded lb # [ 95.567377] niks3-server[1016]: 2026/09/22 16:55:16 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 95.550181] nix-grpc-daemon[857]: time=2026-09-22T16:55:16.197Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 95.551803] nix-grpc-daemon[857]: time=2026-09-22T16:55:16.197Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)" lb # [ 95.599594] niks3-server[1016]: 2026/09/22 16:55:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign worker2 # [ 95.580831] nix-grpc-daemon[857]: time=2026-09-22T16:55:16.227Z level=INFO msg="Uploading 1 narinfos" lb # [ 95.616131] niks3-server[1016]: 2026/09/22 16:55:16 INFO Signed narinfos id=23 count=1 lb # [ 95.622803] niks3-server[1016]: 2026/09/22 16:55:16 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls lb # [ 95.634806] niks3-server[1016]: 2026/09/22 16:55:16 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst worker2 # [ 95.614931] nix-grpc-daemon[857]: time=2026-09-22T16:55:16.262Z level=INFO msg="Upload complete. (241ms)" lb # [ 95.643588] niks3-server[1016]: 2026/09/22 16:55:16 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete lb # [ 95.652374] niks3-server[1016]: 2026/09/22 16:55:16 INFO Completed upload id=23 worker2 # [ 95.628381] nix-grpc-daemon[774]: ts=2026-09-22T16:55:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 95.655035] niks3-server[1016]: 2026/09/22 16:55:16 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo worker2 # [ 95.642938] nix-grpc-daemon[774]: ts=2026-09-22T16:55:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 95.650807] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 95.655770] nix-daemon[843]: reaped child process 1130, status = succeeded worker2 # [ 95.668598] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 96.507535] nix-daemon[843]: reaped child process 1139, status = succeeded worker2 # [ 96.625495] nix-daemon[843]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) worker2 # [ 96.641581] nix-daemon[843]: reaped child process 1158, status = succeeded lb # [ 96.677181] niks3-server[1016]: 2026/09/22 16:55:17 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 96.660670] nix-grpc-daemon[857]: time=2026-09-22T16:55:17.306Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 96.662117] nix-grpc-daemon[857]: time=2026-09-22T16:55:17.306Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)" lb # [ 96.720555] niks3-server[1016]: 2026/09/22 16:55:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign worker2 # [ 96.702846] nix-grpc-daemon[857]: time=2026-09-22T16:55:17.349Z level=INFO msg="Uploading 1 narinfos" lb # [ 96.736366] niks3-server[1016]: 2026/09/22 16:55:17 INFO Signed narinfos id=24 count=1 lb # [ 96.744463] niks3-server[1016]: 2026/09/22 16:55:17 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst lb # [ 96.753702] niks3-server[1016]: 2026/09/22 16:55:17 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls lb # [ 96.764889] niks3-server[1016]: 2026/09/22 16:55:17 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv lb # [ 96.769994] niks3-server[1016]: 2026/09/22 16:55:17 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete worker2 # [ 96.742648] nix-grpc-daemon[857]: time=2026-09-22T16:55:17.389Z level=INFO msg="Upload complete. (261ms)" lb # [ 96.773508] niks3-server[1016]: 2026/09/22 16:55:17 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo lb # [ 96.777609] niks3-server[1016]: 2026/09/22 16:55:17 INFO Completed upload id=24 worker2 # [ 96.754516] nix-grpc-daemon[774]: ts=2026-09-22T16:55:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=13 outputs=1 worker2 # [ 96.773817] nix-daemon[843]: reaped child process 1137, status = succeeded worker1 # [ 96.853273] nix-grpc-daemon[767]: ts=2026-09-22T16:55:17Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 96.875245] nix-grpc-daemon[767]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'... worker1 # [ 96.893018] nix-grpc-daemon[767]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'... worker2 # [ 96.878410] nix-grpc-daemon[774]: ts=2026-09-22T16:55:17Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 96.885317] nix-grpc-daemon[774]: ts=2026-09-22T16:55:17Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=0 paths=1 nar_bytes_out=176 worker1 # [ 97.173252] nix-grpc-daemon[767]: ts=2026-09-22T16:55:17Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 97.180888] nix-grpc-daemon[767]: ts=2026-09-22T16:55:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 97.188375] nix-grpc-daemon[902]: time=2026-09-22T16:55:17.821Z level=INFO msg="All 2 paths already cached" worker1 # [ 97.200150] nix-grpc-daemon[767]: ts=2026-09-22T16:55:17Z level=debug event=want client=13 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=0 worker1 # [ 97.205902] nix-grpc-daemon[767]: ts=2026-09-22T16:55:17Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 97.194004] nix-grpc-daemon[774]: ts=2026-09-22T16:55:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 97.209907] nix-grpc-daemon[774]: ts=2026-09-22T16:55:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 97.226477] nix-grpc-daemon[857]: time=2026-09-22T16:55:17.873Z level=INFO msg="All 2 paths already cached" worker2 # [ 97.236758] nix-grpc-daemon[774]: ts=2026-09-22T16:55:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 97.250891] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 97.413263] nix-daemon[843]: accepted connection from pid 1162, user nix-grpc-daemon (trusted) worker2 # [ 97.433409] nix-daemon[843]: reaped child process 1165, status = succeeded lb # [ 97.459136] niks3-server[1016]: 2026/09/22 16:55:18 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 97.441499] nix-grpc-daemon[857]: time=2026-09-22T16:55:18.088Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 97.442955] nix-grpc-daemon[857]: time=2026-09-22T16:55:18.088Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)" worker2 # [ 97.460262] nix-grpc-daemon[857]: time=2026-09-22T16:55:18.107Z level=INFO msg="Uploading 1 narinfos" lb # [ 97.487805] niks3-server[1016]: 2026/09/22 16:55:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign lb # [ 97.500291] niks3-server[1016]: 2026/09/22 16:55:18 INFO Signed narinfos id=25 count=1 lb # [ 97.503314] niks3-server[1016]: 2026/09/22 16:55:18 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls lb # [ 97.517582] niks3-server[1016]: 2026/09/22 16:55:18 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst worker2 # [ 97.501936] nix-grpc-daemon[857]: time=2026-09-22T16:55:18.149Z level=INFO msg="Upload complete. (233ms)" lb # [ 97.530021] niks3-server[1016]: 2026/09/22 16:55:18 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete lb # [ 97.535832] niks3-server[1016]: 2026/09/22 16:55:18 INFO Completed upload id=25 worker2 # [ 97.512279] nix-grpc-daemon[774]: ts=2026-09-22T16:55:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 97.541371] niks3-server[1016]: 2026/09/22 16:55:18 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo worker2 # [ 97.528148] nix-grpc-daemon[774]: ts=2026-09-22T16:55:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 97.534425] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 97.539464] nix-daemon[843]: reaped child process 1160, status = succeeded worker2 # [ 97.550999] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 98.790016] nix-daemon[843]: reaped child process 1169, status = succeeded worker2 # [ 98.903199] nix-daemon[843]: accepted connection from pid 1185, user nix-grpc-daemon (trusted) worker2 # [ 98.920764] nix-daemon[843]: reaped child process 1188, status = succeeded lb # [ 98.955700] niks3-server[1016]: 2026/09/22 16:55:19 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 98.938461] nix-grpc-daemon[857]: time=2026-09-22T16:55:19.584Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 98.940292] nix-grpc-daemon[857]: time=2026-09-22T16:55:19.584Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)" lb # [ 98.995942] niks3-server[1016]: 2026/09/22 16:55:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign worker2 # [ 98.976472] nix-grpc-daemon[857]: time=2026-09-22T16:55:19.623Z level=INFO msg="Uploading 1 narinfos" lb # [ 99.011262] niks3-server[1016]: 2026/09/22 16:55:19 INFO Signed narinfos id=26 count=1 lb # [ 99.017480] niks3-server[1016]: 2026/09/22 16:55:19 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker2 # [ 99.001530] nix-grpc-daemon[857]: time=2026-09-22T16:55:19.648Z level=INFO msg="Upload complete. (238ms)" lb # [ 99.028356] niks3-server[1016]: 2026/09/22 16:55:19 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst lb # [ 99.036606] niks3-server[1016]: 2026/09/22 16:55:19 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls lb # [ 99.040270] niks3-server[1016]: 2026/09/22 16:55:19 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete lb # [ 99.043818] niks3-server[1016]: 2026/09/22 16:55:19 INFO Completed upload id=26 worker2 # [ 99.012975] nix-grpc-daemon[774]: ts=2026-09-22T16:55:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=14 outputs=1 lb # [ 99.046400] niks3-server[1016]: 2026/09/22 16:55:19 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo worker2 # [ 99.029059] nix-daemon[843]: reaped child process 1167, status = succeeded worker1 # [ 99.078653] nix-grpc-daemon[767]: ts=2026-09-22T16:55:19Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 99.086569] nix-grpc-daemon[767]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'... worker1 # [ 99.122618] nix-grpc-daemon[767]: ts=2026-09-22T16:55:19Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 99.129771] nix-grpc-daemon[767]: ts=2026-09-22T16:55:19Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=0 paths=1 nar_bytes_out=232 lb # [ 99.147436] envoy[895]: conn peer=[2001:db8:1::1]:36442 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3003 tx=811 ms=4304 lb # [ 99.152718] envoy[895]: conn peer=[2001:db8:1::1]:36446 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1605 tx=1599 ms=4849 lb # [ 99.158387] envoy[895]: conn peer=[2001:db8:1::1]:55588 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3445 tx=1943 ms=2243 lb # [ 99.164654] envoy[895]: conn peer=[2001:db8:1::1]:55590 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=72 lb # [ 99.169522] envoy[895]: conn peer=[2001:db8:1::1]:55620 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=375 ms=27 lb # [ 99.174790] envoy[895]: conn peer=[2001:db8:1::1]:55604 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3900 tx=2155 ms=1690 client # [ 99.202791] nix-daemon[1156]: reaped child process 1158, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 6.90 seconds) client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.04 seconds) client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds) client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds) (finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.03 seconds) subtest: an input only one worker has still reaches the builder client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only client # this derivation will be built: client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.30 seconds) client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.12 seconds) worker1: must succeed: nix-store --import < /tmp/shared/inp.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.19 seconds) worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.03 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2 client # [ 100.163985] nix-daemon[1156]: accepted connection from pid 1239, user root (trusted) client # this derivation will be built: client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker2 # [ 100.324875] nix-grpc-daemon[774]: ts=2026-09-22T16:55:20Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:40630 worker1 # [ 100.381537] nix-grpc-daemon[767]: ts=2026-09-22T16:55:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 100.546411] nix-daemon[851]: accepted connection from pid 1128, user nix-grpc-daemon (trusted) worker1 # [ 100.565336] nix-daemon[851]: reaped child process 1131, status = succeeded lb # [ 100.546509] niks3-server[1016]: 2026/09/22 16:55:21 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 100.581662] nix-grpc-daemon[902]: time=2026-09-22T16:55:21.216Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 100.583118] nix-grpc-daemon[902]: time=2026-09-22T16:55:21.216Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)" lb # [ 100.579646] niks3-server[1016]: 2026/09/22 16:55:21 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls worker1 # [ 100.615315] nix-grpc-daemon[902]: time=2026-09-22T16:55:21.250Z level=INFO msg="Uploading 1 narinfos" lb # [ 100.592574] niks3-server[1016]: 2026/09/22 16:55:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign lb # [ 100.603428] niks3-server[1016]: 2026/09/22 16:55:21 INFO Signed narinfos id=27 count=1 lb # [ 100.609629] niks3-server[1016]: 2026/09/22 16:55:21 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst lb # [ 100.622009] niks3-server[1016]: 2026/09/22 16:55:21 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete lb # [ 100.628283] niks3-server[1016]: 2026/09/22 16:55:21 INFO Completed upload id=27 worker1 # [ 100.651475] nix-grpc-daemon[902]: time=2026-09-22T16:55:21.286Z level=INFO msg="Upload complete. (270ms)" lb # [ 100.630488] niks3-server[1016]: 2026/09/22 16:55:21 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo worker1 # [ 100.676001] nix-grpc-daemon[767]: ts=2026-09-22T16:55:21Z level=debug event=want client=14 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=0 worker1 # [ 100.685869] nix-grpc-daemon[767]: ts=2026-09-22T16:55:21Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 100.693615] nix-grpc-daemon[767]: ts=2026-09-22T16:55:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 100.703713] nix-grpc-daemon[767]: ts=2026-09-22T16:55:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:36696 client # copying 2 paths to 'lb:50051'... client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'... worker1 # [ 100.721819] nix-grpc-daemon[767]: ts=2026-09-22T16:55:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'... worker1 # [ 100.735564] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 100.883962] nix-daemon[851]: accepted connection from pid 1137, user nix-grpc-daemon (trusted) worker1 # [ 100.906185] nix-daemon[851]: reaped child process 1140, status = succeeded lb # [ 100.885941] niks3-server[1016]: 2026/09/22 16:55:21 INFO Received uploads request method=POST path=/api/pending_closures lb # [ 100.895933] niks3-server[1016]: 2026/09/22 16:55:21 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 100.930146] nix-grpc-daemon[902]: time=2026-09-22T16:55:21.563Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker1 # [ 100.931638] nix-grpc-daemon[902]: time=2026-09-22T16:55:21.563Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)" worker1 # [ 100.933926] nix-grpc-daemon[902]: time=2026-09-22T16:55:21.564Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)" lb # [ 100.927833] niks3-server[1016]: 2026/09/22 16:55:21 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst lb # [ 100.949418] niks3-server[1016]: 2026/09/22 16:55:21 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst lb # [ 100.959940] niks3-server[1016]: 2026/09/22 16:55:21 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls worker1 # [ 100.994481] nix-grpc-daemon[902]: time=2026-09-22T16:55:21.628Z level=INFO msg="Uploading 3 narinfos" lb # [ 100.971048] niks3-server[1016]: 2026/09/22 16:55:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign lb # [ 100.979904] niks3-server[1016]: 2026/09/22 16:55:21 INFO Signed narinfos id=28 count=1 lb # [ 100.984535] niks3-server[1016]: 2026/09/22 16:55:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign lb # [ 100.987698] niks3-server[1016]: 2026/09/22 16:55:21 INFO Signed narinfos id=29 count=2 lb # [ 100.997376] niks3-server[1016]: 2026/09/22 16:55:21 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls lb # [ 101.009427] niks3-server[1016]: 2026/09/22 16:55:21 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo lb # [ 101.017455] niks3-server[1016]: 2026/09/22 16:55:21 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete lb # [ 101.023320] niks3-server[1016]: 2026/09/22 16:55:21 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo lb # [ 101.029523] niks3-server[1016]: 2026/09/22 16:55:21 INFO Completed upload id=28 worker1 # [ 101.054042] nix-grpc-daemon[902]: time=2026-09-22T16:55:21.689Z level=INFO msg="Upload complete. (301ms)" lb # [ 101.031438] niks3-server[1016]: 2026/09/22 16:55:21 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete lb # [ 101.034739] niks3-server[1016]: 2026/09/22 16:55:21 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo lb # [ 101.038732] niks3-server[1016]: 2026/09/22 16:55:21 INFO Completed upload id=29 worker1 # [ 101.064197] nix-grpc-daemon[767]: ts=2026-09-22T16:55:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=0 paths=2 nar_bytes_in=984 worker1 # [ 101.081486] nix-grpc-daemon[767]: ts=2026-09-22T16:55:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 101.095243] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 101.103928] nix-daemon[851]: reaped child process 1135, status = succeeded client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051... client # uses-input-w1onlya> node-a: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker1 # [ 101.121462] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 102.361169] nix-daemon[851]: reaped child process 1144, status = succeeded worker1 # [ 102.468220] nix-daemon[851]: accepted connection from pid 1160, user nix-grpc-daemon (trusted) worker1 # [ 102.483371] nix-daemon[851]: reaped child process 1163, status = succeeded lb # [ 102.468539] niks3-server[1016]: 2026/09/22 16:55:23 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 102.502366] nix-grpc-daemon[902]: time=2026-09-22T16:55:23.135Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 102.507746] niks3-server[1016]: 2026/09/22 16:55:23 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker1 # [ 102.541526] nix-grpc-daemon[902]: time=2026-09-22T16:55:23.176Z level=INFO msg="Uploading 1 narinfos" lb # [ 102.522584] niks3-server[1016]: 2026/09/22 16:55:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign lb # [ 102.531009] niks3-server[1016]: 2026/09/22 16:55:23 INFO Signed narinfos id=30 count=1 lb # [ 102.536964] niks3-server[1016]: 2026/09/22 16:55:23 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls worker1 # [ 102.573374] nix-grpc-daemon[902]: time=2026-09-22T16:55:23.208Z level=INFO msg="Upload complete. (238ms)" lb # [ 102.549688] niks3-server[1016]: 2026/09/22 16:55:23 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete lb # [ 102.554456] niks3-server[1016]: 2026/09/22 16:55:23 INFO Completed upload id=30 lb # [ 102.562655] niks3-server[1016]: 2026/09/22 16:55:23 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo worker1 # [ 102.589081] nix-grpc-daemon[767]: ts=2026-09-22T16:55:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=15 outputs=1 worker1 # [ 102.603845] nix-daemon[851]: reaped child process 1142, status = succeeded client # copying 1 paths... worker2 # [ 102.583875] nix-grpc-daemon[774]: ts=2026-09-22T16:55:23Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 102.599274] nix-grpc-daemon[774]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'... client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'... worker2 # [ 102.644149] nix-grpc-daemon[774]: ts=2026-09-22T16:55:23Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 102.651845] nix-grpc-daemon[774]: ts=2026-09-22T16:55:23Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=0 paths=1 nar_bytes_out=120 client # [ 102.768295] nix-daemon[1156]: reaped child process 1242, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.80 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2 client # [ 102.942573] nix-daemon[1156]: accepted connection from pid 1261, user root (trusted) client # this derivation will be built: client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker2 # [ 103.102370] nix-grpc-daemon[774]: ts=2026-09-22T16:55:23Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:40630 client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker2 # [ 103.110266] nix-grpc-daemon[774]: ts=2026-09-22T16:55:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 worker1 # [ 103.181555] nix-grpc-daemon[767]: ts=2026-09-22T16:55:23Z level=debug event=want client=15 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=0 worker2 # [ 103.133260] nix-grpc-daemon[774]: ts=2026-09-22T16:55:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker1 # [ 103.187935] nix-grpc-daemon[767]: ts=2026-09-22T16:55:23Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 103.147402] nix-grpc-daemon[774]: ts=2026-09-22T16:55:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40630 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'... worker2 # [ 103.168413] nix-grpc-daemon[774]: ts=2026-09-22T16:55:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 103.183535] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 103.205228] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 98 ms (attempt 1/5) worker2 # [ 103.311241] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 176 ms (attempt 2/5) worker2 # [ 103.489429] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 219 ms (attempt 3/5) worker2 # [ 103.719342] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 788 ms (attempt 4/5) worker2 # [ 104.509108] nix-grpc-daemon[774]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 104.509518] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 104.521461] nix-grpc-daemon[774]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'... worker2 # [ 104.709733] nix-daemon[843]: accepted connection from pid 1201, user nix-grpc-daemon (trusted) worker2 # [ 104.730366] nix-daemon[843]: reaped child process 1204, status = succeeded lb # [ 104.763870] niks3-server[1016]: 2026/09/22 16:55:25 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 104.746982] nix-grpc-daemon[857]: time=2026-09-22T16:55:25.393Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 104.748783] nix-grpc-daemon[857]: time=2026-09-22T16:55:25.393Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)" lb # [ 104.794291] niks3-server[1016]: 2026/09/22 16:55:25 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls worker2 # [ 104.778190] nix-grpc-daemon[857]: time=2026-09-22T16:55:25.425Z level=INFO msg="Uploading 1 narinfos" lb # [ 104.807742] niks3-server[1016]: 2026/09/22 16:55:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign lb # [ 104.818795] niks3-server[1016]: 2026/09/22 16:55:25 INFO Signed narinfos id=31 count=1 worker2 # [ 104.801142] nix-grpc-daemon[857]: time=2026-09-22T16:55:25.448Z level=INFO msg="Upload complete. (255ms)" lb # [ 104.829872] niks3-server[1016]: 2026/09/22 16:55:25 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst lb # [ 104.836462] niks3-server[1016]: 2026/09/22 16:55:25 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete lb # [ 104.842718] niks3-server[1016]: 2026/09/22 16:55:25 INFO Completed upload id=31 lb # [ 104.845387] niks3-server[1016]: 2026/09/22 16:55:25 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo worker2 # [ 104.812987] nix-grpc-daemon[774]: ts=2026-09-22T16:55:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=1 paths=1 nar_bytes_in=568 worker2 # [ 104.826971] nix-grpc-daemon[774]: ts=2026-09-22T16:55:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 104.832711] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) worker2 # [ 104.837559] nix-daemon[843]: reaped child process 1194, status = succeeded worker2 # [ 104.845978] nix-grpc-daemon[774]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'... worker2 # [ 104.860243] nix-daemon[843]: accepted connection from pid 774, user nix-grpc-daemon (trusted) client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051... client # uses-input-w1onlyb> worker2: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 105.927331] nix-daemon[843]: reaped child process 1209, status = succeeded worker2 # [ 106.060204] nix-daemon[843]: accepted connection from pid 1225, user nix-grpc-daemon (trusted) worker2 # [ 106.075521] nix-daemon[843]: reaped child process 1228, status = succeeded lb # [ 106.111871] niks3-server[1016]: 2026/09/22 16:55:26 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 106.096470] nix-grpc-daemon[857]: time=2026-09-22T16:55:26.740Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 106.142109] niks3-server[1016]: 2026/09/22 16:55:26 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls lb # [ 106.154044] niks3-server[1016]: 2026/09/22 16:55:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign worker2 # [ 106.132452] nix-grpc-daemon[857]: time=2026-09-22T16:55:26.779Z level=INFO msg="Uploading 1 narinfos" lb # [ 106.167250] niks3-server[1016]: 2026/09/22 16:55:26 INFO Signed narinfos id=32 count=1 lb # [ 106.172803] niks3-server[1016]: 2026/09/22 16:55:26 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv lb # [ 106.184855] niks3-server[1016]: 2026/09/22 16:55:26 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete lb # [ 106.187902] niks3-server[1016]: 2026/09/22 16:55:26 INFO Completed upload id=32 lb # [ 106.191816] niks3-server[1016]: 2026/09/22 16:55:26 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo worker2 # [ 106.155707] nix-grpc-daemon[857]: time=2026-09-22T16:55:26.802Z level=INFO msg="Upload complete. (278ms)" client # copying 1 paths... worker2 # [ 106.169356] nix-grpc-daemon[774]: ts=2026-09-22T16:55:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40630 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=16 outputs=1 worker2 # [ 106.187163] nix-grpc-daemon[774]: ts=2026-09-22T16:55:26Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:40630 worker2 # [ 106.193060] nix-daemon[843]: reaped child process 1206, status = succeeded client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'... worker1 # [ 106.272950] nix-grpc-daemon[767]: ts=2026-09-22T16:55:26Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 106.286945] nix-grpc-daemon[767]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'... worker1 # [ 106.304142] nix-grpc-daemon[767]: ts=2026-09-22T16:55:26Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=0 paths=1 nar_bytes_out=120 client # [ 106.363283] nix-daemon[1156]: reaped child process 1264, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.59 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2 client # [ 106.534941] nix-daemon[1156]: accepted connection from pid 1283, user root (trusted) client # this derivation will be built: client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker1 # [ 106.747980] nix-grpc-daemon[767]: ts=2026-09-22T16:55:27Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:36696 client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 106.755637] nix-grpc-daemon[767]: ts=2026-09-22T16:55:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 106.763931] nix-grpc-daemon[902]: time=2026-09-22T16:55:27.396Z level=INFO msg="All 1 paths already cached" worker1 # [ 106.775185] nix-grpc-daemon[767]: ts=2026-09-22T16:55:27Z level=debug event=want client=16 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=0 worker1 # [ 106.784269] nix-grpc-daemon[767]: ts=2026-09-22T16:55:27Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 106.792850] nix-grpc-daemon[767]: ts=2026-09-22T16:55:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 106.800263] nix-grpc-daemon[767]: ts=2026-09-22T16:55:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 106.810353] nix-grpc-daemon[902]: time=2026-09-22T16:55:27.445Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'... worker1 # [ 106.820782] nix-grpc-daemon[767]: ts=2026-09-22T16:55:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 106.837068] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 106.982959] nix-daemon[851]: accepted connection from pid 1172, user nix-grpc-daemon (trusted) worker1 # [ 107.001594] nix-daemon[851]: reaped child process 1175, status = succeeded lb # [ 106.981976] niks3-server[1016]: 2026/09/22 16:55:27 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 107.015603] nix-grpc-daemon[902]: time=2026-09-22T16:55:27.650Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 107.017289] nix-grpc-daemon[902]: time=2026-09-22T16:55:27.650Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)" lb # [ 107.011184] niks3-server[1016]: 2026/09/22 16:55:27 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls worker1 # [ 107.043988] nix-grpc-daemon[902]: time=2026-09-22T16:55:27.679Z level=INFO msg="Uploading 1 narinfos" lb # [ 107.024613] niks3-server[1016]: 2026/09/22 16:55:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign lb # [ 107.036406] niks3-server[1016]: 2026/09/22 16:55:27 INFO Signed narinfos id=33 count=1 worker1 # [ 107.072505] nix-grpc-daemon[902]: time=2026-09-22T16:55:27.707Z level=INFO msg="Upload complete. (221ms)" lb # [ 107.049255] niks3-server[1016]: 2026/09/22 16:55:27 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst lb # [ 107.058123] niks3-server[1016]: 2026/09/22 16:55:27 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete worker1 # [ 107.080234] nix-grpc-daemon[767]: ts=2026-09-22T16:55:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=0 paths=1 nar_bytes_in=568 lb # [ 107.065713] niks3-server[1016]: 2026/09/22 16:55:27 INFO Completed upload id=33 lb # [ 107.068700] niks3-server[1016]: 2026/09/22 16:55:27 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo worker1 # [ 107.095644] nix-grpc-daemon[767]: ts=2026-09-22T16:55:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 107.102459] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 107.107384] nix-daemon[851]: reaped child process 1170, status = succeeded client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051... worker1 # [ 107.122086] nix-daemon[851]: accepted connection from pid 767, user nix-grpc-daemon (trusted) client # uses-input-w1onlyc> node-a: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 108.192290] nix-daemon[851]: reaped child process 1179, status = succeeded worker1 # [ 108.284144] nix-daemon[851]: accepted connection from pid 1195, user nix-grpc-daemon (trusted) worker1 # [ 108.305155] nix-daemon[851]: reaped child process 1198, status = succeeded lb # [ 108.279599] niks3-server[1016]: 2026/09/22 16:55:28 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 108.316405] nix-grpc-daemon[902]: time=2026-09-22T16:55:28.950Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 108.313750] niks3-server[1016]: 2026/09/22 16:55:28 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls worker1 # [ 108.349292] nix-grpc-daemon[902]: time=2026-09-22T16:55:28.984Z level=INFO msg="Uploading 1 narinfos" lb # [ 108.326085] niks3-server[1016]: 2026/09/22 16:55:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign lb # [ 108.335023] niks3-server[1016]: 2026/09/22 16:55:28 INFO Signed narinfos id=34 count=1 lb # [ 108.342461] niks3-server[1016]: 2026/09/22 16:55:28 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker1 # [ 108.375249] nix-grpc-daemon[902]: time=2026-09-22T16:55:29.010Z level=INFO msg="Upload complete. (243ms)" lb # [ 108.352703] niks3-server[1016]: 2026/09/22 16:55:29 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete lb # [ 108.360205] niks3-server[1016]: 2026/09/22 16:55:29 INFO Completed upload id=34 lb # [ 108.362327] niks3-server[1016]: 2026/09/22 16:55:29 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo worker1 # [ 108.384710] nix-grpc-daemon[767]: ts=2026-09-22T16:55:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=17 outputs=1 worker1 # [ 108.400264] nix-daemon[851]: reaped child process 1177, status = succeeded client # copying 1 paths... worker1 # [ 108.433736] nix-grpc-daemon[767]: ts=2026-09-22T16:55:29Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:36696 client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'... worker1 # [ 108.464840] nix-grpc-daemon[767]: ts=2026-09-22T16:55:29Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 108.471266] nix-grpc-daemon[767]: ts=2026-09-22T16:55:29Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36696 duration_s=0 paths=1 nar_bytes_out=120 client # [ 108.535605] nix-daemon[1156]: reaped child process 1286, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.17 seconds) (finished: subtest: an input only one worker has still reaches the builder, in 9.20 seconds) subtest: upload whose reference the worker lacks is completed from the cache client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache client # this derivation will be built: client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.33 seconds) client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache client # this derivation will be built: client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv lb # [ 109.146220] envoy[895]: conn peer=[2001:db8:1::1]:55632 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4338 tx=2392 ms=2025 lb # [ 109.151288] envoy[895]: conn peer=[2001:db8:1::1]:55636 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=406 ms=79 lb # [ 109.156351] envoy[895]: conn peer=[2001:db8:1::1]:55646 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3874 tx=2068 ms=2387 lb # [ 109.161057] envoy[895]: conn peer=[2001:db8:1::1]:37512 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=69 lb # [ 109.165501] envoy[895]: conn peer=[2001:db8:1::1]:37514 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3744 tx=2127 ms=3207 lb # [ 109.170249] envoy[895]: conn peer=[2001:db8:1::1]:37530 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=329 ms=94 lb # [ 109.175651] envoy[895]: conn peer=[2001:db8:1::1]:37540 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3783 tx=2099 ms=1781 client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'... lb # [ 109.180510] envoy[895]: conn peer=[2001:db8:1::1]:37554 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=68 client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.28 seconds) client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.12 seconds) worker1: must succeed: nix-store --import < /tmp/shared/ref.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.20 seconds) worker2 # [ 109.551013] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 110.490617] nix-grpc-daemon[774]: ts=2026-09-22T16:55:31Z level=info event=signal_exit worker1 # [ 112.339315] nix-grpc-daemon[767]: ts=2026-09-22T16:55:32Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:45386 worker2 # [ 112.290142] nix-grpc-daemon[774]: ts=2026-09-22T16:55:32Z level=info event=scheduler_restarting worker2 # [ 112.319695] nix-grpc-daemon[774]: ts=2026-09-22T16:55:32Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 112.345689] nix-daemon[843]: reaped child process 845, status = succeeded worker2 # [ 112.666742] nix-grpc-daemon[774]: E0000 00:00:1790096133.313697 819 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true worker2 # [ 112.673349] nix-grpc-daemon[774]: E0000 00:00:1790096133.313996 819 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker1 # [ 112.344419] nix-grpc-daemon[767]: ts=2026-09-22T16:55:32Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 112.965714] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 112.970204] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 112.976443] systemd[1]: nix-grpc-daemon.service: Consumed 3.720s CPU time over 1min 33.722s wall clock time, 62M memory peak, 104.8K incoming IP traffic, 254K outgoing IP traffic. worker2 # [ 112.992473] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker2 # [ 113.004815] systemd[1]: Closed Nix worker-protocol over gRPC. worker2 # [ 113.006697] systemd[1]: nix-grpc-daemon.socket: Consumed 763us CPU time over 1min 43.507s wall clock time, 256K memory peak, 106.3K incoming IP traffic, 90.1K outgoing IP traffic. lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds) client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2 worker1 # [ 113.350786] nix-grpc-daemon[767]: ts=2026-09-22T16:55:33Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:36696 worker1 # [ 113.359669] nix-grpc-daemon[767]: ts=2026-09-22T16:55:33Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:36696 client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.25 seconds) worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds) worker2 # [ 113.436328] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 113.566521] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 113.963359] systemd[1]: Starting scheduler-token.service... worker2 # [ 114.049883] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 114.056711] systemd[1]: Finished scheduler-token.service. worker2 # [ 114.058704] systemd[1]: scheduler-token.service: Consumed 56ms CPU time over 89ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 114.069770] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 114.203594] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:34Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 114.206064] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:34Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 114.209868] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:34Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 114.215445] nix-grpc-daemon[1252]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 114.233896] nix-grpc-daemon[1252]: E0000 00:00:1790096134.861939 1252 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 114.263787] nix-grpc-daemon[1252]: E0000 00:00:1790096134.865588 1252 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 114.268254] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:34Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 114.270380] nix-grpc-daemon[1252]: E0000 00:00:1790096134.877650 1292 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 114.272996] nix-grpc-daemon[1252]: E0000 00:00:1790096134.897270 1292 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 114.275287] nix-grpc-daemon[1252]: E0000 00:00:1790096134.897417 1292 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 114.280490] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 114.281375] nix-grpc-daemon[1252]: E0000 00:00:1790096134.914658 1292 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 114.285660] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 114.337268] nix-grpc-daemon[767]: ts=2026-09-22T16:55:34Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:45386 worker1 # [ 114.343233] nix-grpc-daemon[767]: ts=2026-09-22T16:55:34Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 114.459528] nix-grpc-daemon[767]: ts=2026-09-22T16:55:35Z level=info event=signal_exit lb # [ 115.314259] niks3-server[1016]: 2026/09/22 16:55:35 INFO lead: released remote=192.168.1.3:52562 worker2 # [ 116.281689] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:36Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 116.335432] nix-grpc-daemon[767]: ts=2026-09-22T16:55:36Z level=info event=scheduler_restarting worker1 # [ 116.346014] nix-grpc-daemon[767]: ts=2026-09-22T16:55:36Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:45386 worker1 # [ 116.351262] nix-grpc-daemon[767]: ts=2026-09-22T16:55:36Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 116.355931] nix-grpc-daemon[767]: ts=2026-09-22T16:55:36Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 116.360663] nix-grpc-daemon[767]: ts=2026-09-22T16:55:36Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:45386 worker1 # [ 116.366292] nix-grpc-daemon[767]: ts=2026-09-22T16:55:36Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 116.392397] nix-daemon[851]: reaped child process 853, status = succeeded worker2 # [ 116.481308] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 116.486537] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 116.568671] nix-grpc-daemon[767]: ts=2026-09-22T16:55:37Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker1 # [ 116.610475] nix-grpc-daemon[767]: E0000 00:00:1790096137.245376 812 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true worker1 # [ 116.617089] nix-grpc-daemon[767]: E0000 00:00:1790096137.245714 812 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker2 # [ 116.687667] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 116.692913] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 116.894092] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 116.899384] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 116.964159] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 116.971478] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 116.977825] systemd[1]: nix-grpc-daemon.service: Consumed 3.000s CPU time over 1min 37.788s wall clock time, 65.9M memory peak, 100.6K incoming IP traffic, 198.2K outgoing IP traffic. worker1 # [ 116.997056] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker1 # [ 116.999140] systemd[1]: Closed Nix worker-protocol over gRPC. lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 117.004902] systemd[1]: nix-grpc-daemon.socket: Consumed 565us CPU time over 1min 46.741s wall clock time, 256K memory peak, 122.9K incoming IP traffic, 105.4K outgoing IP traffic. lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds) client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2 worker2 # [ 117.100472] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 117.105770] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 117.258853] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 117.273685] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) worker2 # [ 117.303029] nix-grpc-daemon[1252]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 10 ms (attempt 1/5) worker2 # [ 117.310549] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 117.314530] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 117.321722] nix-grpc-daemon[1252]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 69 ms (attempt 2/5) worker2 # [ 117.398216] nix-grpc-daemon[1252]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 89 ms (attempt 3/5) worker2 # [ 117.488835] nix-grpc-daemon[1252]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 757 ms (attempt 4/5) worker2 # [ 117.515601] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 117.520935] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 117.722042] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 117.727071] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 117.928299] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 117.933345] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 118.134314] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 118.139403] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 118.249454] nix-grpc-daemon[1252]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 118.256461] nix-grpc-daemon[1252]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org client # copying 2 paths... client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'... worker2 # [ 118.279759] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37496 client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'... worker2 # [ 118.298433] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) worker2 # [ 118.329067] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:38Z level=info event=niks3_push_started pid=1312 worker2 # [ 118.341332] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 118.356650] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 118.495884] nix-grpc-daemon[1312]: time=2026-09-22T16:55:39.140Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 118.548998] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 118.551781] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 118.604078] nix-daemon[843]: accepted connection from pid 1325, user nix-grpc-daemon (trusted) worker2 # [ 118.626351] nix-daemon[843]: reaped child process 1328, status = succeeded lb # [ 118.652988] niks3-server[1016]: 2026/09/22 16:55:39 INFO Received uploads request method=POST path=/api/pending_closures lb # [ 118.663687] niks3-server[1016]: 2026/09/22 16:55:39 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 118.647055] nix-grpc-daemon[1312]: time=2026-09-22T16:55:39.292Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker2 # [ 118.650944] nix-grpc-daemon[1312]: time=2026-09-22T16:55:39.292Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)" worker2 # [ 118.653330] nix-grpc-daemon[1312]: time=2026-09-22T16:55:39.292Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)" lb # [ 118.722436] niks3-server[1016]: 2026/09/22 16:55:39 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls lb # [ 118.739083] niks3-server[1016]: 2026/09/22 16:55:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign lb # [ 118.747699] niks3-server[1016]: 2026/09/22 16:55:39 INFO Signed narinfos id=35 count=1 worker2 # [ 118.719355] nix-grpc-daemon[1312]: time=2026-09-22T16:55:39.364Z level=INFO msg="Uploading 3 narinfos" lb # [ 118.752711] niks3-server[1016]: 2026/09/22 16:55:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign lb # [ 118.755159] niks3-server[1016]: 2026/09/22 16:55:39 INFO Signed narinfos id=36 count=2 lb # [ 118.763175] niks3-server[1016]: 2026/09/22 16:55:39 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst lb # [ 118.768511] niks3-server[1016]: 2026/09/22 16:55:39 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls lb # [ 118.770548] niks3-server[1016]: 2026/09/22 16:55:39 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst worker2 # [ 118.752388] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:39Z level=info event=scheduler_connected addr=lb:50051 lb # [ 118.783441] niks3-server[1016]: 2026/09/22 16:55:39 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo worker2 # [ 118.758040] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 118.790794] niks3-server[1016]: 2026/09/22 16:55:39 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete lb # [ 118.795788] niks3-server[1016]: 2026/09/22 16:55:39 INFO Completed upload id=35 lb # [ 118.798560] niks3-server[1016]: 2026/09/22 16:55:39 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete worker2 # [ 118.771233] nix-grpc-daemon[1312]: time=2026-09-22T16:55:39.418Z level=INFO msg="Upload complete. (276ms)" lb # [ 118.800727] niks3-server[1016]: 2026/09/22 16:55:39 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo lb # [ 118.802881] niks3-server[1016]: 2026/09/22 16:55:39 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo lb # [ 118.804926] niks3-server[1016]: 2026/09/22 16:55:39 INFO Completed upload id=36 worker2 # [ 118.784603] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37496 duration_s=0 paths=2 nar_bytes_in=312 worker2 # [ 118.800645] nix-daemon[843]: reaped child process 1310, status = succeeded client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.79 seconds) worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds) worker1 # [ 118.975125] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 118.986275] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 118.959200] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 118.964671] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 119.135317] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:39Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 119.137771] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:39Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 119.141778] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:39Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 119.147236] nix-grpc-daemon[1221]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR lb # [ 119.132563] niks3-server[1016]: 2026/09/22 16:55:39 INFO lead: acquired remote=192.168.1.3:60652 worker1 # [ 119.168208] nix-grpc-daemon[1221]: E0000 00:00:1790096139.781952 1221 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 119.170582] nix-grpc-daemon[1221]: E0000 00:00:1790096139.785689 1221 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb # [ 119.147730] envoy[895]: conn peer=[2001:db8:1::4]:57320 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=76823 tx=4257 ms=92790 lb # [ 119.153115] envoy[895]: conn peer=[2001:db8:1::1]:40944 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=31 lb # [ 119.157783] envoy[895]: conn peer=[2001:db8:1::3]:41134 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=15122 tx=4082 ms=97072 lb # [ 119.163114] envoy[895]: conn peer=[2001:db8:1::1]:40960 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1004 tx=305 ms=1540 lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 119.196981] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:39Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 119.199132] nix-grpc-daemon[1221]: E0000 00:00:1790096139.792545 1263 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 119.201864] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:39Z level=info event=scheduler_take_over worker1 # [ 119.202971] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 119.203743] nix-grpc-daemon[1221]: E0000 00:00:1790096139.814687 1263 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 119.206198] nix-grpc-daemon[1221]: E0000 00:00:1790096139.814892 1263 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 119.211664] nix-grpc-daemon[1221]: E0000 00:00:1790096139.831440 1263 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 119.216053] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 119.217787] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 119.165932] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 119.171116] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker1 # [ 119.418891] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 119.423887] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 119.373790] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 119.377994] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 119.624565] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 119.574551] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 119.629870] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 119.579812] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 119.830728] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 119.780843] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 119.836151] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 119.785950] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 120.036754] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 119.987108] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 120.045579] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:40Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:52898 worker1 # [ 120.052480] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:40Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 120.064124] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:40Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 120.068726] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:40Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:52898 worker1 # [ 120.073764] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:40Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.15 seconds) (finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.73 seconds) subtest: a niks3 restart does not cost the leader its role lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb # [ 120.438654] systemd[1]: Stopping niks3 server... lb # [ 120.441814] niks3-server[1016]: 2026/09/22 16:55:41 INFO Shutdown signal received, draining in-flight requests timeout=10s worker1 # [ 120.471472] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:41Z level=info event=scheduler_lock_lost status=200 worker2 # [ 120.419472] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:41Z level=info event=scheduler_lock_lost status=200 lb # [ 120.465004] niks3-server[1016]: 2026/09/22 16:55:41 INFO lead: released remote=192.168.1.3:60652 lb # [ 120.470081] systemd[1]: niks3.service: Deactivated successfully. lb # [ 120.472363] systemd[1]: Stopped niks3 server. lb # [ 120.473149] systemd[1]: niks3.service: Consumed 658ms CPU time over 1min 31.396s wall clock time, 27.8M memory peak, 100.1K incoming IP traffic, 143.1K outgoing IP traffic. lb # [ 120.483586] systemd[1]: Starting niks3 server... lb # [ 120.549411] niks3-server[1154]: 2026/09/22 16:55:41 goose: no migrations to run. current version: 20260920000000 lb # [ 120.557077] niks3-server[1154]: 2026/09/22 16:55:41 OK 1_commit_pending_closure.sql (5.73ms) lb # [ 120.560660] niks3-server[1154]: 2026/09/22 16:55:41 OK 2_object_stats_trigger.sql (3.21ms) lb # [ 120.561813] niks3-server[1154]: 2026/09/22 16:55:41 goose: up to current file version: 2 lb # [ 120.563449] niks3-server[1154]: 2026/09/22 16:55:41 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key lb # [ 120.574246] niks3-server[1154]: 2026/09/22 16:55:41 INFO Read proxy enabled — serving cache objects from S3 lb # [ 120.578892] niks3-server[1154]: 2026/09/22 16:55:41 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 120.581874] niks3-server[1154]: 2026/09/22 16:55:41 INFO systemd watchdog enabled interval=15s lb # [ 120.584415] niks3-server[1154]: 2026/09/22 16:55:41 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 120.585805] systemd[1]: Started niks3 server. lb: waiting for unit niks3.service lb: (finished: waiting for unit niks3.service, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield worker1: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.07 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2 worker1 # [ 121.090168] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:43244 worker1 # [ 121.101744] nix-daemon[851]: accepted connection from pid 1221, user nix-grpc-daemon (trusted) client # don't know how to build these paths: client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker2 # [ 121.077912] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:37496 worker1 # [ 121.163159] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:41Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=0 worker1 # [ 121.170355] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:41Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 121.216984] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43244 worker1 # [ 121.236976] nix-grpc-daemon[1221]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 72 ms (attempt 1/5) worker1 # [ 121.317954] nix-grpc-daemon[1221]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 21 ms (attempt 2/5) worker1 # [ 121.340955] nix-grpc-daemon[1221]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 258 ms (attempt 3/5) lb # [ 121.452155] niks3-server[1154]: 2026/09/22 16:55:42 INFO lead: acquired remote=192.168.1.3:37990 worker1 # [ 121.609529] nix-grpc-daemon[1221]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 504 ms (attempt 4/5) worker1 # [ 122.115408] nix-grpc-daemon[1221]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 122.121410] nix-grpc-daemon[1221]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 122.140275] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43244 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'... worker1 # [ 122.158996] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43244 worker1 # [ 122.174364] nix-daemon[851]: accepted connection from pid 1221, user nix-grpc-daemon (trusted) worker1 # [ 122.209791] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:42Z level=info event=niks3_push_started pid=1293 worker1 # [ 122.358882] nix-grpc-daemon[1293]: time=2026-09-22T16:55:42.993Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 122.484234] nix-daemon[851]: accepted connection from pid 1305, user nix-grpc-daemon (trusted) worker1 # [ 122.503763] nix-daemon[851]: reaped child process 1308, status = succeeded lb # [ 122.485998] niks3-server[1154]: 2026/09/22 16:55:43 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 122.523751] nix-grpc-daemon[1293]: time=2026-09-22T16:55:43.157Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 122.527719] nix-grpc-daemon[1293]: time=2026-09-22T16:55:43.157Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)" lb # [ 122.543807] niks3-server[1154]: 2026/09/22 16:55:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign lb # [ 122.558942] niks3-server[1154]: 2026/09/22 16:55:43 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls lb # [ 122.581003] niks3-server[1154]: 2026/09/22 16:55:43 INFO Signed narinfos id=37 count=1 worker1 # [ 122.609952] nix-grpc-daemon[1293]: time=2026-09-22T16:55:43.243Z level=INFO msg="Uploading 1 narinfos" lb # [ 122.600177] niks3-server[1154]: 2026/09/22 16:55:43 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst lb # [ 122.603238] niks3-server[1154]: 2026/09/22 16:55:43 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete lb # [ 122.612810] niks3-server[1154]: 2026/09/22 16:55:43 INFO Completed upload id=37 worker1 # [ 122.637747] nix-grpc-daemon[1293]: time=2026-09-22T16:55:43.273Z level=INFO msg="Upload complete. (275ms)" lb # [ 122.618484] niks3-server[1154]: 2026/09/22 16:55:43 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo worker1 # [ 122.649147] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43244 duration_s=0 paths=1 nar_bytes_in=544 worker1 # [ 122.668987] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43244 worker1 # [ 122.677902] nix-daemon[851]: accepted connection from pid 1221, user nix-grpc-daemon (trusted) worker1 # [ 122.685170] nix-daemon[851]: reaped child process 1291, status = succeeded client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051... client # farm-a> node-a: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker1 # [ 122.701860] nix-daemon[851]: accepted connection from pid 1221, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-after-niks3-restart worker1 # [ 123.611716] nix-daemon[851]: reaped child process 1312, status = succeeded worker1 # [ 123.682812] nix-daemon[851]: accepted connection from pid 1328, user nix-grpc-daemon (trusted) worker1 # [ 123.705204] nix-daemon[851]: reaped child process 1331, status = succeeded lb # [ 123.682064] niks3-server[1154]: 2026/09/22 16:55:44 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 123.719626] nix-grpc-daemon[1293]: time=2026-09-22T16:55:44.354Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 123.724613] nix-grpc-daemon[1293]: time=2026-09-22T16:55:44.356Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)" lb # [ 123.721862] niks3-server[1154]: 2026/09/22 16:55:44 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv lb # [ 123.742049] niks3-server[1154]: 2026/09/22 16:55:44 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst worker1 # [ 123.779152] nix-grpc-daemon[1293]: time=2026-09-22T16:55:44.413Z level=INFO msg="Uploading 1 narinfos" lb # [ 123.759096] niks3-server[1154]: 2026/09/22 16:55:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign lb # [ 123.768776] niks3-server[1154]: 2026/09/22 16:55:44 INFO Signed narinfos id=38 count=1 lb # [ 123.770887] niks3-server[1154]: 2026/09/22 16:55:44 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls worker1 # [ 123.802908] nix-grpc-daemon[1293]: time=2026-09-22T16:55:44.438Z level=INFO msg="Upload complete. (288ms)" lb # [ 123.778574] niks3-server[1154]: 2026/09/22 16:55:44 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete lb # [ 123.783452] niks3-server[1154]: 2026/09/22 16:55:44 INFO Completed upload id=38 lb # [ 123.788476] niks3-server[1154]: 2026/09/22 16:55:44 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo worker1 # [ 123.824992] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43244 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=1 worker1 # [ 123.840644] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:44Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=0 worker1 # [ 123.846742] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:44Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 123.851035] nix-daemon[851]: reaped child process 1310, status = succeeded worker1 # [ 123.870010] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43244 worker1 # [ 123.886428] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43244 worker1 # [ 123.901312] nix-grpc-daemon[1293]: time=2026-09-22T16:55:44.536Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'... worker1 # [ 123.912454] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43244 worker1 # [ 123.926344] nix-daemon[851]: accepted connection from pid 1221, user nix-grpc-daemon (trusted) worker1 # [ 124.084505] nix-daemon[851]: accepted connection from pid 1337, user nix-grpc-daemon (trusted) worker1 # [ 124.103386] nix-daemon[851]: reaped child process 1340, status = succeeded lb # [ 124.083190] niks3-server[1154]: 2026/09/22 16:55:44 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 124.119935] nix-grpc-daemon[1293]: time=2026-09-22T16:55:44.754Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 124.122251] nix-grpc-daemon[1293]: time=2026-09-22T16:55:44.754Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)" lb # [ 124.116251] niks3-server[1154]: 2026/09/22 16:55:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign worker1 # [ 124.150412] nix-grpc-daemon[1293]: time=2026-09-22T16:55:44.785Z level=INFO msg="Uploading 1 narinfos" lb # [ 124.133890] niks3-server[1154]: 2026/09/22 16:55:44 INFO Signed narinfos id=39 count=1 lb # [ 124.142846] niks3-server[1154]: 2026/09/22 16:55:44 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls lb # [ 124.154340] niks3-server[1154]: 2026/09/22 16:55:44 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst worker1 # [ 124.191951] nix-grpc-daemon[1293]: time=2026-09-22T16:55:44.827Z level=INFO msg="Upload complete. (249ms)" lb # [ 124.168131] niks3-server[1154]: 2026/09/22 16:55:44 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete lb # [ 124.171795] niks3-server[1154]: 2026/09/22 16:55:44 INFO Completed upload id=39 lb # [ 124.175836] niks3-server[1154]: 2026/09/22 16:55:44 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo worker1 # [ 124.204151] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43244 duration_s=0 paths=1 nar_bytes_in=656 worker1 # [ 124.220707] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43244 worker1 # [ 124.227965] nix-daemon[851]: accepted connection from pid 1221, user nix-grpc-daemon (trusted) worker1 # [ 124.234108] nix-daemon[851]: reaped child process 1335, status = succeeded client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051... worker1 # [ 124.246158] nix-daemon[851]: accepted connection from pid 1221, user nix-grpc-daemon (trusted) client # farm-b> node-a: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-after-niks3-restart worker1 # [ 125.482636] nix-daemon[851]: reaped child process 1344, status = succeeded worker1 # [ 125.591079] nix-daemon[851]: accepted connection from pid 1360, user nix-grpc-daemon (trusted) worker1 # [ 125.610723] nix-daemon[851]: reaped child process 1363, status = succeeded lb # [ 125.591994] niks3-server[1154]: 2026/09/22 16:55:46 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 125.627416] nix-grpc-daemon[1293]: time=2026-09-22T16:55:46.261Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 125.629149] nix-grpc-daemon[1293]: time=2026-09-22T16:55:46.261Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)" lb # [ 125.636455] niks3-server[1154]: 2026/09/22 16:55:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign worker1 # [ 125.671827] nix-grpc-daemon[1293]: time=2026-09-22T16:55:46.306Z level=INFO msg="Uploading 1 narinfos" lb # [ 125.650710] niks3-server[1154]: 2026/09/22 16:55:46 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst lb # [ 125.662837] niks3-server[1154]: 2026/09/22 16:55:46 INFO Signed narinfos id=40 count=1 lb # [ 125.672319] niks3-server[1154]: 2026/09/22 16:55:46 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker1 # [ 125.702507] nix-grpc-daemon[1293]: time=2026-09-22T16:55:46.337Z level=INFO msg="Upload complete. (244ms)" lb # [ 125.675986] niks3-server[1154]: 2026/09/22 16:55:46 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls lb # [ 125.683850] niks3-server[1154]: 2026/09/22 16:55:46 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete lb # [ 125.690065] niks3-server[1154]: 2026/09/22 16:55:46 INFO Completed upload id=40 lb # [ 125.692551] niks3-server[1154]: 2026/09/22 16:55:46 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo worker1 # [ 125.720386] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43244 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=1 worker1 # [ 125.736970] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:46Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=0 worker1 # [ 125.742548] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:46Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 125.747225] nix-daemon[851]: reaped child process 1342, status = succeeded worker1 # [ 125.777631] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43244 worker1 # [ 125.793264] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43244 worker1 # [ 125.807001] nix-grpc-daemon[1293]: time=2026-09-22T16:55:46.442Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'... worker1 # [ 125.818719] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43244 worker1 # [ 125.833106] nix-daemon[851]: accepted connection from pid 1221, user nix-grpc-daemon (trusted) worker1 # [ 125.989689] nix-daemon[851]: accepted connection from pid 1369, user nix-grpc-daemon (trusted) worker1 # [ 126.008315] nix-daemon[851]: reaped child process 1372, status = succeeded lb # [ 125.990545] niks3-server[1154]: 2026/09/22 16:55:46 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 126.024801] nix-grpc-daemon[1293]: time=2026-09-22T16:55:46.659Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 126.026654] nix-grpc-daemon[1293]: time=2026-09-22T16:55:46.659Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)" lb # [ 126.021886] niks3-server[1154]: 2026/09/22 16:55:46 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls lb # [ 126.032724] niks3-server[1154]: 2026/09/22 16:55:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign worker1 # [ 126.065214] nix-grpc-daemon[1293]: time=2026-09-22T16:55:46.700Z level=INFO msg="Uploading 1 narinfos" lb # [ 126.045941] niks3-server[1154]: 2026/09/22 16:55:46 INFO Signed narinfos id=41 count=1 lb # [ 126.051688] niks3-server[1154]: 2026/09/22 16:55:46 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst worker1 # [ 126.087202] nix-grpc-daemon[1293]: time=2026-09-22T16:55:46.722Z level=INFO msg="Upload complete. (238ms)" lb # [ 126.064748] niks3-server[1154]: 2026/09/22 16:55:46 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete lb # [ 126.069853] niks3-server[1154]: 2026/09/22 16:55:46 INFO Completed upload id=41 worker1 # [ 126.096257] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43244 duration_s=0 paths=1 nar_bytes_in=784 lb # [ 126.073763] niks3-server[1154]: 2026/09/22 16:55:46 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo worker1 # [ 126.108496] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43244 worker1 # [ 126.118471] nix-daemon[851]: accepted connection from pid 1221, user nix-grpc-daemon (trusted) worker1 # [ 126.124773] nix-daemon[851]: reaped child process 1367, status = succeeded client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051... worker1 # [ 126.137782] nix-daemon[851]: accepted connection from pid 1221, user nix-grpc-daemon (trusted) client # farm-top> node-a: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-after-niks3-restart worker1 # [ 127.242301] nix-daemon[851]: reaped child process 1376, status = succeeded worker1 # [ 127.312987] nix-daemon[851]: accepted connection from pid 1392, user nix-grpc-daemon (trusted) worker1 # [ 127.330158] nix-daemon[851]: reaped child process 1395, status = succeeded lb # [ 127.313922] niks3-server[1154]: 2026/09/22 16:55:47 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 127.348428] nix-grpc-daemon[1293]: time=2026-09-22T16:55:47.983Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 127.350001] nix-grpc-daemon[1293]: time=2026-09-22T16:55:47.983Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)" lb # [ 127.355266] niks3-server[1154]: 2026/09/22 16:55:48 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls worker1 # [ 127.392720] nix-grpc-daemon[1293]: time=2026-09-22T16:55:48.027Z level=INFO msg="Uploading 1 narinfos" lb # [ 127.367706] niks3-server[1154]: 2026/09/22 16:55:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign lb # [ 127.380895] niks3-server[1154]: 2026/09/22 16:55:48 INFO Signed narinfos id=42 count=1 lb # [ 127.383643] niks3-server[1154]: 2026/09/22 16:55:48 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv lb # [ 127.399663] niks3-server[1154]: 2026/09/22 16:55:48 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete worker1 # [ 127.429756] nix-grpc-daemon[1293]: time=2026-09-22T16:55:48.065Z level=INFO msg="Upload complete. (281ms)" lb # [ 127.410044] niks3-server[1154]: 2026/09/22 16:55:48 INFO Completed upload id=42 lb # [ 127.414409] niks3-server[1154]: 2026/09/22 16:55:48 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst lb # [ 127.421822] niks3-server[1154]: 2026/09/22 16:55:48 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo worker1 # [ 127.445360] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43244 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=1 worker1 # [ 127.463440] nix-daemon[851]: reaped child process 1374, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2, in 6.62 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath, in 0.19 seconds) (finished: subtest: a niks3 restart does not cost the leader its role, in 7.34 seconds) subtest: scheduler down: the other node takes the lock and keeps it lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1 # [ 127.927929] systemd[1]: Stopping Nix worker-protocol over gRPC... worker1 # [ 128.159189] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:48Z level=info event=signal_exit lb # [ 128.456580] niks3-server[1154]: 2026/09/22 16:55:49 INFO lead: released remote=192.168.1.3:37990 lb # [ 129.148939] envoy[895]: conn peer=[2001:db8:1::1]:57674 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9723 tx=4419 ms=6380 worker1 # [ 129.482215] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:50Z level=info event=scheduler_restarting worker2 # [ 129.432913] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:50Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 129.495403] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:52898 worker1 # [ 129.500564] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:50Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 129.505142] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:50Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 129.510291] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:52898 worker1 # [ 129.520562] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:50Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 129.538146] nix-daemon[851]: reaped child process 1281, status = succeeded worker2 # [ 129.632972] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:50Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 129.638175] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 129.711978] nix-grpc-daemon[1221]: ts=2026-09-22T16:55:50Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 129.839314] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:50Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 129.845015] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 129.997971] nix-grpc-daemon[1221]: E0000 00:00:1790096150.633176 1266 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 10, started true, shutdown true worker1 # [ 130.006104] nix-grpc-daemon[1221]: E0000 00:00:1790096150.641340 1266 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker2 # [ 130.046056] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:50Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 130.051564] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 130.252838] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:50Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 130.257937] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 130.461098] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:51Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 130.465335] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 130.661855] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:51Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 130.667229] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 130.761450] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 130.768839] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 130.771416] systemd[1]: nix-grpc-daemon.service: Consumed 1.255s CPU time over 11.778s wall clock time, 38.1M memory peak, 36.4K incoming IP traffic, 59.1K outgoing IP traffic. worker1 # [ 130.788430] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker1 # [ 130.790640] systemd[1]: Closed Nix worker-protocol over gRPC. worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over worker1 # [ 130.799804] systemd[1]: nix-grpc-daemon.socket: Consumed 1ms CPU time over 11.823s wall clock time, 256K memory peak, 33.4K incoming IP traffic, 24.6K outgoing IP traffic. worker2 # [ 130.868454] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:51Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 130.873620] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 131.074775] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:51Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 131.080384] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 131.281081] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:51Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 131.287280] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 131.425925] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:52Z level=info event=scheduler_take_over lb # [ 131.454237] niks3-server[1154]: 2026/09/22 16:55:52 INFO lead: acquired remote=192.168.1.4:60234 worker2 # [ 131.490277] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:52Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 131.494521] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 131.691626] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:52Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 131.697089] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 131.898626] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:52Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 131.905112] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 1.17 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker2 # [ 132.104255] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:52Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 132.109846] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 132.317443] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:52Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 132.330554] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 132.531351] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:53Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 132.560175] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:53Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 132.565322] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:53Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:47028 worker2 # [ 132.570723] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:53Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2 client # don't know how to build these paths: client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker2 # [ 133.241746] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 133.249891] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 133.276138] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:53Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=0 worker2 # [ 133.282357] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:53Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 133.329093] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 133.345280] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:37496 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'... worker2 # [ 133.361053] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 133.375927] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) worker2 # [ 133.533304] nix-daemon[843]: accepted connection from pid 1357, user nix-grpc-daemon (trusted) worker2 # [ 133.553392] nix-daemon[843]: reaped child process 1360, status = succeeded lb # [ 133.586093] niks3-server[1154]: 2026/09/22 16:55:54 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 133.571744] nix-grpc-daemon[1312]: time=2026-09-22T16:55:54.218Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 133.573674] nix-grpc-daemon[1312]: time=2026-09-22T16:55:54.218Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)" lb # [ 133.618107] niks3-server[1154]: 2026/09/22 16:55:54 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls worker2 # [ 133.600186] nix-grpc-daemon[1312]: time=2026-09-22T16:55:54.247Z level=INFO msg="Uploading 1 narinfos" lb # [ 133.628800] niks3-server[1154]: 2026/09/22 16:55:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign lb # [ 133.637138] niks3-server[1154]: 2026/09/22 16:55:54 INFO Signed narinfos id=43 count=1 lb # [ 133.647699] niks3-server[1154]: 2026/09/22 16:55:54 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst lb # [ 133.661821] niks3-server[1154]: 2026/09/22 16:55:54 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete worker2 # [ 133.643385] nix-grpc-daemon[1312]: time=2026-09-22T16:55:54.290Z level=INFO msg="Upload complete. (250ms)" lb # [ 133.673774] niks3-server[1154]: 2026/09/22 16:55:54 INFO Completed upload id=43 lb # [ 133.676308] niks3-server[1154]: 2026/09/22 16:55:54 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo worker2 # [ 133.657381] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37496 duration_s=0 paths=1 nar_bytes_in=520 worker2 # [ 133.672881] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 133.681596] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) worker2 # [ 133.687085] nix-daemon[843]: reaped child process 1355, status = succeeded client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051... worker2 # [ 133.698242] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) client # farm-a> worker2: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-on-standby worker2 # [ 135.158909] nix-daemon[843]: reaped child process 1364, status = succeeded worker2 # [ 135.277793] nix-daemon[843]: accepted connection from pid 1380, user nix-grpc-daemon (trusted) worker2 # [ 135.293598] nix-daemon[843]: reaped child process 1383, status = succeeded lb # [ 135.330767] niks3-server[1154]: 2026/09/22 16:55:55 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 135.316468] nix-grpc-daemon[1312]: time=2026-09-22T16:55:55.961Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 135.318260] nix-grpc-daemon[1312]: time=2026-09-22T16:55:55.961Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)" lb # [ 135.381898] niks3-server[1154]: 2026/09/22 16:55:56 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv lb # [ 135.392195] niks3-server[1154]: 2026/09/22 16:55:56 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst worker2 # [ 135.372849] nix-grpc-daemon[1312]: time=2026-09-22T16:55:56.019Z level=INFO msg="Uploading 1 narinfos" lb # [ 135.404852] niks3-server[1154]: 2026/09/22 16:55:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign lb # [ 135.413284] niks3-server[1154]: 2026/09/22 16:55:56 INFO Signed narinfos id=44 count=1 lb # [ 135.415685] niks3-server[1154]: 2026/09/22 16:55:56 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls lb # [ 135.428489] niks3-server[1154]: 2026/09/22 16:55:56 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete worker2 # [ 135.400421] nix-grpc-daemon[1312]: time=2026-09-22T16:55:56.047Z level=INFO msg="Upload complete. (289ms)" lb # [ 135.432439] niks3-server[1154]: 2026/09/22 16:55:56 INFO Completed upload id=44 lb # [ 135.435632] niks3-server[1154]: 2026/09/22 16:55:56 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo worker2 # [ 135.422273] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37496 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=1 worker2 # [ 135.439269] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:56Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=0 worker2 # [ 135.443510] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:56Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 135.447786] nix-daemon[843]: reaped child process 1362, status = succeeded worker2 # [ 135.465370] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 135.480657] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 135.495709] nix-grpc-daemon[1312]: time=2026-09-22T16:55:56.142Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'... worker2 # [ 135.506644] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 135.520918] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) worker2 # [ 135.671047] nix-daemon[843]: accepted connection from pid 1389, user nix-grpc-daemon (trusted) worker2 # [ 135.689690] nix-daemon[843]: reaped child process 1392, status = succeeded lb # [ 135.720991] niks3-server[1154]: 2026/09/22 16:55:56 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 135.703480] nix-grpc-daemon[1312]: time=2026-09-22T16:55:56.350Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 135.705276] nix-grpc-daemon[1312]: time=2026-09-22T16:55:56.350Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)" lb # [ 135.751140] niks3-server[1154]: 2026/09/22 16:55:56 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls worker2 # [ 135.735720] nix-grpc-daemon[1312]: time=2026-09-22T16:55:56.382Z level=INFO msg="Uploading 1 narinfos" lb # [ 135.766556] niks3-server[1154]: 2026/09/22 16:55:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign lb # [ 135.776463] niks3-server[1154]: 2026/09/22 16:55:56 INFO Signed narinfos id=45 count=1 lb # [ 135.781571] niks3-server[1154]: 2026/09/22 16:55:56 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst lb # [ 135.790666] niks3-server[1154]: 2026/09/22 16:55:56 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete worker2 # [ 135.769337] nix-grpc-daemon[1312]: time=2026-09-22T16:55:56.416Z level=INFO msg="Upload complete. (232ms)" lb # [ 135.799665] niks3-server[1154]: 2026/09/22 16:55:56 INFO Completed upload id=45 lb # [ 135.802138] niks3-server[1154]: 2026/09/22 16:55:56 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo worker2 # [ 135.779818] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37496 duration_s=0 paths=1 nar_bytes_in=640 worker2 # [ 135.796840] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 135.804992] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) worker2 # [ 135.810573] nix-daemon[843]: reaped child process 1387, status = succeeded client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051... worker2 # [ 135.823801] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) client # farm-b> worker2: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-on-standby worker2 # [ 136.935596] nix-daemon[843]: reaped child process 1396, status = succeeded worker2 # [ 137.009800] nix-daemon[843]: accepted connection from pid 1412, user nix-grpc-daemon (trusted) worker2 # [ 137.027451] nix-daemon[843]: reaped child process 1415, status = succeeded lb # [ 137.061891] niks3-server[1154]: 2026/09/22 16:55:57 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 137.045501] nix-grpc-daemon[1312]: time=2026-09-22T16:55:57.691Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 137.047754] nix-grpc-daemon[1312]: time=2026-09-22T16:55:57.691Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)" lb # [ 137.101005] niks3-server[1154]: 2026/09/22 16:55:57 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst worker2 # [ 137.086191] nix-grpc-daemon[1312]: time=2026-09-22T16:55:57.732Z level=INFO msg="Uploading 1 narinfos" lb # [ 137.116534] niks3-server[1154]: 2026/09/22 16:55:57 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv lb # [ 137.125633] niks3-server[1154]: 2026/09/22 16:55:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign lb # [ 137.144454] niks3-server[1154]: 2026/09/22 16:55:57 INFO Signed narinfos id=46 count=1 lb # [ 137.153310] niks3-server[1154]: 2026/09/22 16:55:57 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete worker2 # [ 137.125885] nix-grpc-daemon[1312]: time=2026-09-22T16:55:57.773Z level=INFO msg="Upload complete. (287ms)" lb # [ 137.161607] niks3-server[1154]: 2026/09/22 16:55:57 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls lb # [ 137.170289] niks3-server[1154]: 2026/09/22 16:55:57 INFO Completed upload id=46 worker2 # [ 137.142150] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37496 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=1 worker2 # [ 137.164320] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:57Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=0 worker2 # [ 137.169947] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:57Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 137.175215] nix-daemon[843]: reaped child process 1394, status = succeeded worker2 # [ 137.205003] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 137.220333] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 137.235897] nix-grpc-daemon[1312]: time=2026-09-22T16:55:57.883Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'... worker2 # [ 137.247074] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 137.261170] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) worker2 # [ 137.412969] nix-daemon[843]: accepted connection from pid 1421, user nix-grpc-daemon (trusted) worker2 # [ 137.432322] nix-daemon[843]: reaped child process 1424, status = succeeded lb # [ 137.465726] niks3-server[1154]: 2026/09/22 16:55:58 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 137.450208] nix-grpc-daemon[1312]: time=2026-09-22T16:55:58.097Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 137.451972] nix-grpc-daemon[1312]: time=2026-09-22T16:55:58.097Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)" lb # [ 137.500481] niks3-server[1154]: 2026/09/22 16:55:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign worker2 # [ 137.481393] nix-grpc-daemon[1312]: time=2026-09-22T16:55:58.128Z level=INFO msg="Uploading 1 narinfos" lb # [ 137.514579] niks3-server[1154]: 2026/09/22 16:55:58 INFO Signed narinfos id=47 count=1 lb # [ 137.521444] niks3-server[1154]: 2026/09/22 16:55:58 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls lb # [ 137.534817] niks3-server[1154]: 2026/09/22 16:55:58 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst lb # [ 137.539878] niks3-server[1154]: 2026/09/22 16:55:58 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete worker2 # [ 137.512209] nix-grpc-daemon[1312]: time=2026-09-22T16:55:58.159Z level=INFO msg="Upload complete. (234ms)" lb # [ 137.543231] niks3-server[1154]: 2026/09/22 16:55:58 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo lb # [ 137.547683] niks3-server[1154]: 2026/09/22 16:55:58 INFO Completed upload id=47 worker2 # [ 137.524049] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37496 duration_s=0 paths=1 nar_bytes_in=768 worker2 # [ 137.532977] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 137.544757] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) worker2 # [ 137.549799] nix-daemon[843]: reaped child process 1419, status = succeeded client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051... client # farm-top> worker2: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker2 # [ 137.564122] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-on-standby worker2 # [ 138.395132] nix-daemon[843]: reaped child process 1428, status = succeeded worker2 # [ 138.497518] nix-daemon[843]: accepted connection from pid 1444, user nix-grpc-daemon (trusted) worker2 # [ 138.514621] nix-daemon[843]: reaped child process 1447, status = succeeded lb # [ 138.548768] niks3-server[1154]: 2026/09/22 16:55:59 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 138.534560] nix-grpc-daemon[1312]: time=2026-09-22T16:55:59.180Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 138.536101] nix-grpc-daemon[1312]: time=2026-09-22T16:55:59.180Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)" lb # [ 138.592734] niks3-server[1154]: 2026/09/22 16:55:59 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker2 # [ 138.574274] nix-grpc-daemon[1312]: time=2026-09-22T16:55:59.221Z level=INFO msg="Uploading 1 narinfos" lb # [ 138.604386] niks3-server[1154]: 2026/09/22 16:55:59 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst lb # [ 138.613952] niks3-server[1154]: 2026/09/22 16:55:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign lb # [ 138.624778] niks3-server[1154]: 2026/09/22 16:55:59 INFO Signed narinfos id=48 count=1 worker2 # [ 138.596247] nix-grpc-daemon[1312]: time=2026-09-22T16:55:59.243Z level=INFO msg="Upload complete. (222ms)" lb # [ 138.627119] niks3-server[1154]: 2026/09/22 16:55:59 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls lb # [ 138.635474] niks3-server[1154]: 2026/09/22 16:55:59 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete worker2 # [ 138.608842] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37496 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=1 lb # [ 138.643338] niks3-server[1154]: 2026/09/22 16:55:59 INFO Completed upload id=48 lb # [ 138.646935] niks3-server[1154]: 2026/09/22 16:55:59 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo worker2 # [ 138.623765] nix-daemon[843]: reaped child process 1426, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 5.63 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.18 seconds) worker1 # [ 138.964895] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 138.971494] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 139.121358] nix-grpc-daemon[1413]: ts=2026-09-22T16:55:59Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 139.123646] nix-grpc-daemon[1413]: ts=2026-09-22T16:55:59Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 139.127437] nix-grpc-daemon[1413]: ts=2026-09-22T16:55:59Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 139.132048] nix-grpc-daemon[1413]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 139.133478] nix-grpc-daemon[1413]: E0000 00:00:1790096159.767371 1413 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 139.146170] nix-grpc-daemon[1413]: E0000 00:00:1790096159.773374 1413 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 139.162356] nix-grpc-daemon[1413]: ts=2026-09-22T16:55:59Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket lb # [ 139.147631] envoy[895]: conn peer=[2001:db8:1::3]:41382 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1605 tx=620 ms=10493 lb # [ 139.155879] envoy[895]: conn peer=[2001:db8:1::1]:33162 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9589 tx=4400 ms=5384 worker1 # [ 139.183457] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 139.184916] nix-grpc-daemon[1413]: E0000 00:00:1790096159.782723 1453 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 139.187403] nix-grpc-daemon[1413]: E0000 00:00:1790096159.802935 1453 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 139.192782] nix-grpc-daemon[1413]: E0000 00:00:1790096159.803089 1453 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 139.197129] nix-grpc-daemon[1413]: E0000 00:00:1790096159.817794 1453 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 139.199430] nix-grpc-daemon[1413]: ts=2026-09-22T16:55:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 139.145835] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:59Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:47028 worker2 # [ 139.151685] nix-grpc-daemon[1252]: ts=2026-09-22T16:55:59Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: scheduler down: the other node takes the lock and keeps it, in 11.78 seconds) subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs client # [ 139.459054] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs. client # Running as unit: rs.service; invocation ID: bde0feca499e45dd9e20a99a185452ba client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.09 seconds) client # [ 139.668156] nix[1410]: don't know how to build these paths: client # [ 139.669225] nix[1410]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker2 # [ 139.679194] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 139.693090] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:00Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=0 worker2 # [ 139.700455] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:00Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 139.740902] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 139.756728] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:37496 client # [ 139.759395] nix[1410]: copying 1 paths to 'lb:50051'... worker2 # [ 139.771312] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37496 client # [ 139.762024] nix[1410]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'... worker2 # [ 139.786784] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) worker2 # [ 139.938975] nix-daemon[843]: accepted connection from pid 1460, user nix-grpc-daemon (trusted) worker2 # [ 139.961464] nix-daemon[843]: reaped child process 1463, status = succeeded lb # [ 139.987573] niks3-server[1154]: 2026/09/22 16:56:00 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 139.969943] nix-grpc-daemon[1312]: time=2026-09-22T16:56:00.616Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 139.971509] nix-grpc-daemon[1312]: time=2026-09-22T16:56:00.616Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)" lb # [ 140.013873] niks3-server[1154]: 2026/09/22 16:56:00 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls lb # [ 140.028999] niks3-server[1154]: 2026/09/22 16:56:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign worker2 # [ 140.007122] nix-grpc-daemon[1312]: time=2026-09-22T16:56:00.654Z level=INFO msg="Uploading 1 narinfos" lb # [ 140.040991] niks3-server[1154]: 2026/09/22 16:56:00 INFO Signed narinfos id=49 count=1 lb # [ 140.046845] niks3-server[1154]: 2026/09/22 16:56:00 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst worker2 # [ 140.033020] nix-grpc-daemon[1312]: time=2026-09-22T16:56:00.680Z level=INFO msg="Upload complete. (229ms)" lb # [ 140.061067] niks3-server[1154]: 2026/09/22 16:56:00 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete lb # [ 140.068537] niks3-server[1154]: 2026/09/22 16:56:00 INFO Completed upload id=49 worker2 # [ 140.044263] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37496 duration_s=0 paths=1 nar_bytes_in=416 lb # [ 140.073697] niks3-server[1154]: 2026/09/22 16:56:00 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo worker2 # [ 140.053022] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37496 worker2 # [ 140.063865] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) worker2 # [ 140.069919] nix-daemon[843]: reaped child process 1458, status = succeeded worker2 # [ 140.079779] nix-daemon[843]: accepted connection from pid 1252, user nix-grpc-daemon (trusted) client # [ 140.066955] nix[1410]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051... lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true worker1 # [ 142.319556] systemd-journald[271]: Received client request to rotate journal, rotating. worker1 # [ 142.342297] systemd-journald[271]: Vacuuming done, freed 0B of archived journals from /var/log/journal/9afbec48643448bf9b5cdf96fc447472. worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.09 seconds) worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service worker2 # [ 142.337771] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds) client: must succeed: sleep 5.0 worker2 # [ 143.236842] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:03Z level=info event=draining client: (finished: must succeed: sleep 5.0, in 5.05 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=' worker2 # [ 147.718273] nix-daemon[843]: reaped child process 1467, status = succeeded worker2 # [ 147.816774] nix-daemon[843]: accepted connection from pid 1508, user nix-grpc-daemon (trusted) worker2 # [ 147.837199] nix-daemon[843]: reaped child process 1511, status = succeeded lb # [ 147.869804] niks3-server[1154]: 2026/09/22 16:56:08 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 147.852575] nix-grpc-daemon[1312]: time=2026-09-22T16:56:08.499Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 147.855874] nix-grpc-daemon[1312]: time=2026-09-22T16:56:08.499Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)" lb # [ 147.915003] niks3-server[1154]: 2026/09/22 16:56:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign worker2 # [ 147.897368] nix-grpc-daemon[1312]: time=2026-09-22T16:56:08.544Z level=INFO msg="Uploading 1 narinfos" lb # [ 147.930381] niks3-server[1154]: 2026/09/22 16:56:08 INFO Signed narinfos id=50 count=1 lb # [ 147.944484] niks3-server[1154]: 2026/09/22 16:56:08 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst lb # [ 147.961393] niks3-server[1154]: 2026/09/22 16:56:08 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls lb # [ 147.968479] niks3-server[1154]: 2026/09/22 16:56:08 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker2 # [ 147.940398] nix-grpc-daemon[1312]: time=2026-09-22T16:56:08.587Z level=INFO msg="Upload complete. (244ms)" lb # [ 147.971612] niks3-server[1154]: 2026/09/22 16:56:08 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete lb # [ 147.976668] niks3-server[1154]: 2026/09/22 16:56:08 INFO Completed upload id=50 lb # [ 147.978674] niks3-server[1154]: 2026/09/22 16:56:08 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo worker2 # [ 147.953098] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37496 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=1 worker2 # [ 147.972697] nix-daemon[843]: reaped child process 1465, status = succeeded client # [ 147.988206] systemd[1]: rs.service: Deactivated successfully. client # [ 147.993253] systemd[1]: rs.service: Consumed 198ms CPU time over 8.532s wall clock time, 41.2M memory peak, 7.3K incoming IP traffic, 8.4K outgoing IP traffic. worker2 # [ 148.237594] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:08Z level=info event=signal_exit lb # [ 148.456744] niks3-server[1154]: 2026/09/22 16:56:09 INFO lead: released remote=192.168.1.4:60234 lb # [ 149.121145] niks3-server[1154]: 2026/09/22 16:56:09 INFO lead: acquired remote=192.168.1.3:54490 worker1 # [ 149.146054] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:09Z level=info event=scheduler_take_over lb # [ 149.146688] envoy[895]: conn peer=[2001:db8:1::1]:33174 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=1542 ms=8302 worker1 # [ 149.482531] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:10Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 149.431148] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:10Z level=info event=scheduler_restarting worker2 # [ 149.440533] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:10Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 149.445024] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:10Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:47028 worker2 # [ 149.449789] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:10Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 149.454048] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:10Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:47028 worker2 # [ 149.458948] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:10Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 149.484316] nix-daemon[843]: reaped child process 1304, status = succeeded worker1 # [ 149.683332] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:10Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 149.689082] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 149.651437] nix-grpc-daemon[1252]: ts=2026-09-22T16:56:10Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker1 # [ 149.889713] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:10Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 149.901495] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:10Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:46294 worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.21 seconds) client: waiting for success: ! systemctl is-active rs worker1 # [ 149.907142] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:10Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds) client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; } client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds) client: must fail: journalctl -u rs | grep -q 'reconnecting' client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker2 # [ 150.355468] nix-grpc-daemon[1252]: E0000 00:00:1790096171.002558 1297 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true worker2 # [ 150.388734] nix-grpc-daemon[1252]: E0000 00:00:1790096171.035867 1297 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2 # [ 151.266456] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker2 # [ 151.272160] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 151.274725] systemd[1]: nix-grpc-daemon.service: Consumed 1.760s CPU time over 37.198s wall clock time, 39.7M memory peak, 61.7K incoming IP traffic, 144.5K outgoing IP traffic. worker2 # [ 151.295974] systemd[1]: Starting scheduler-token.service... worker2 # [ 151.370441] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 151.374960] systemd[1]: Finished scheduler-token.service. worker2 # [ 151.380834] systemd[1]: scheduler-token.service: Consumed 46ms CPU time over 77ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 151.387400] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 151.518971] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:12Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 151.524103] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:12Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 151.526033] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:12Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 151.552158] nix-grpc-daemon[1521]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 151.553564] nix-grpc-daemon[1521]: E0000 00:00:1790096172.176995 1521 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 151.555803] nix-grpc-daemon[1521]: E0000 00:00:1790096172.179017 1521 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 151.575104] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:12Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 151.577323] nix-grpc-daemon[1521]: E0000 00:00:1790096172.185391 1560 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 151.579819] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 151.580712] nix-grpc-daemon[1521]: E0000 00:00:1790096172.207027 1560 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 151.586000] nix-grpc-daemon[1521]: E0000 00:00:1790096172.207176 1560 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 151.590347] nix-grpc-daemon[1521]: E0000 00:00:1790096172.221584 1560 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 151.592995] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:12Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 151.651987] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:12Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 151.654101] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:12Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:46294 worker1 # [ 151.654234] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:12Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.99 seconds) subtest: scheduler restart mid-build: build finishes, no double build client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid client # [ 152.453569] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid. client # Running as unit: mid.service; invocation ID: ca709b7867f348d4b51242b4dce26c5a client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.09 seconds) worker2 # [ 152.675423] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:43910 worker2 # [ 152.689584] nix-daemon[843]: accepted connection from pid 1521, user nix-grpc-daemon (trusted) client # [ 152.695128] nix[1446]: don't know how to build these paths: client # [ 152.697034] nix[1446]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker1 # [ 152.778264] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:13Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker1 # [ 152.778567] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:13Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 152.836043] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50018 worker1 # [ 152.851005] nix-daemon[851]: accepted connection from pid 1413, user nix-grpc-daemon (trusted) worker1 # [ 152.875332] nix-grpc-daemon[1413]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 26 ms (attempt 1/5) worker1 # [ 152.909830] nix-grpc-daemon[1413]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 73 ms (attempt 2/5) worker1 # [ 152.984722] nix-grpc-daemon[1413]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 138 ms (attempt 3/5) worker1 # [ 153.132100] nix-grpc-daemon[1413]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 142 ms (attempt 4/5) worker1 # [ 153.276192] nix-grpc-daemon[1413]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 153.282454] nix-grpc-daemon[1413]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 153.301469] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50018 client # [ 153.250498] nix[1446]: copying 1 paths to 'lb:50051'... client # [ 153.255907] nix[1446]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'... worker1 # [ 153.321103] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50018 worker1 # [ 153.335101] nix-daemon[851]: accepted connection from pid 1413, user nix-grpc-daemon (trusted) worker1 # [ 153.372237] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:13Z level=info event=niks3_push_started pid=1538 worker1 # [ 153.519222] nix-grpc-daemon[1538]: time=2026-09-22T16:56:14.154Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 153.626174] nix-daemon[851]: accepted connection from pid 1551, user nix-grpc-daemon (trusted) worker1 # [ 153.641639] nix-daemon[851]: reaped child process 1554, status = succeeded lb # [ 153.625984] niks3-server[1154]: 2026/09/22 16:56:14 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 153.660240] nix-grpc-daemon[1538]: time=2026-09-22T16:56:14.295Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 153.666316] nix-grpc-daemon[1538]: time=2026-09-22T16:56:14.298Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)" lb # [ 153.665769] niks3-server[1154]: 2026/09/22 16:56:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign worker1 # [ 153.703042] nix-grpc-daemon[1538]: time=2026-09-22T16:56:14.337Z level=INFO msg="Uploading 1 narinfos" lb # [ 153.681464] niks3-server[1154]: 2026/09/22 16:56:14 INFO Signed narinfos id=51 count=1 lb # [ 153.691510] niks3-server[1154]: 2026/09/22 16:56:14 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst lb # [ 153.709527] niks3-server[1154]: 2026/09/22 16:56:14 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete worker1 # [ 153.741416] nix-grpc-daemon[1538]: time=2026-09-22T16:56:14.376Z level=INFO msg="Upload complete. (218ms)" worker1 # [ 153.748102] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50018 duration_s=0 paths=1 nar_bytes_in=424 lb # [ 153.725662] niks3-server[1154]: 2026/09/22 16:56:14 INFO Completed upload id=51 lb # [ 153.729073] niks3-server[1154]: 2026/09/22 16:56:14 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo worker1 # [ 153.757870] nix-grpc-daemon[1413]: ts=2026-09-22T16:56:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50018 worker1 # [ 153.762235] nix-daemon[851]: accepted connection from pid 1413, user nix-grpc-daemon (trusted) lb # [ 153.736972] niks3-server[1154]: 2026/09/22 16:56:14 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls worker1 # [ 153.766063] nix-daemon[851]: reaped child process 1536, status = succeeded worker1 # [ 153.774105] nix-daemon[851]: accepted connection from pid 1413, user nix-grpc-daemon (trusted) client # [ 153.710637] nix[1446]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service worker1 # [ 155.207104] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1413 (nix-grpc-daemon) on client request. worker1 # [ 155.209352] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request. lb # [ 155.186206] niks3-server[1154]: 2026/09/22 16:56:15 INFO lead: released remote=192.168.1.3:54490 client # [ 155.153993] nix[1446]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again worker2 # [ 155.166198] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:15Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination" worker1 # [ 155.227200] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL worker1: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.06 seconds) client: must succeed: sleep 5.0 worker1 # [ 155.232511] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'. worker1 # [ 155.233744] systemd[1]: nix-grpc-daemon.service: Consumed 519ms CPU time over 16.255s wall clock time, 34.5M memory peak, 15.5K incoming IP traffic, 26.6K outgoing IP traffic. worker1 # [ 155.258333] nix-daemon[851]: reaped child process 1560, status = succeeded worker1 # [ 155.259762] nix-daemon[851]: reaped child process 1530, status = succeeded worker1 # [ 155.268054] nix-daemon[1562]: unexpected Nix daemon error: error: interrupted by the user worker1 # [ 155.271855] nix-daemon[851]: reaped child process 1562, status = failed with exit code 1 worker2 # [ 155.366651] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 155.422795] systemd[1]: nix-grpc-daemon.service: Scheduled restart job, restart counter is at 1. worker1 # [ 155.434765] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 155.596469] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:16Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 155.599226] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:16Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 155.603256] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:16Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 155.608052] nix-grpc-daemon[1595]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 155.609535] nix-grpc-daemon[1595]: E0000 00:00:1790096176.243354 1595 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 155.601225] niks3-server[1154]: 2026/09/22 16:56:16 INFO lead: acquired remote=192.168.1.3:54268 worker1 # [ 155.652055] nix-grpc-daemon[1595]: E0000 00:00:1790096176.249530 1595 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 155.654331] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:16Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 155.655941] nix-grpc-daemon[1595]: E0000 00:00:1790096176.257895 1634 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 155.663167] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:16Z level=info event=scheduler_take_over worker1 # [ 155.664784] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:16Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 155.666165] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:16Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:50026 worker1 # [ 155.667840] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:16Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 155.672390] nix-grpc-daemon[1595]: E0000 00:00:1790096176.276916 1634 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 155.676704] nix-grpc-daemon[1595]: E0000 00:00:1790096176.277033 1634 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 155.679552] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 155.680435] nix-grpc-daemon[1595]: E0000 00:00:1790096176.297840 1634 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 155.682665] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 155.683910] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:16Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:50026 worker1 # [ 155.685536] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:16Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 155.726961] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:16Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker1 # [ 155.730463] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:16Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 155.721508] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43910 worker2 # [ 155.738264] nix-grpc-daemon[1521]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 34 ms (attempt 1/5) worker2 # [ 155.781683] nix-grpc-daemon[1521]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 162 ms (attempt 2/5) client # [ 155.160429] nix[1446]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting worker2 # [ 155.953025] nix-grpc-daemon[1521]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 194 ms (attempt 3/5) worker2 # [ 156.156313] nix-grpc-daemon[1521]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 57 ms (attempt 4/5) worker2 # [ 156.217344] nix-grpc-daemon[1521]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 156.222898] nix-grpc-daemon[1521]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 156.239747] nix-grpc-daemon[1521]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' from 'http://lb:5751'... worker2 # [ 156.268473] nix-daemon[843]: accepted connection from pid 1521, user nix-grpc-daemon (trusted) client # [ 156.274522] nix[1446]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... worker2 # [ 156.289072] nix-daemon[843]: accepted connection from pid 1521, user nix-grpc-daemon (trusted) lb # [ 159.148226] envoy[895]: conn peer=[2001:db8:1::4]:39572 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=27145 tx=1735 ms=35361 lb # [ 159.153391] envoy[895]: conn peer=[2001:db8:1::3]:41878 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=801 tx=376 ms=16030 client: (finished: must succeed: sleep 5.0, in 5.05 seconds) client: waiting for success: ! systemctl is-active mid worker2 # [ 160.513735] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:21Z level=info event=niks3_push_started pid=1623 worker2 # [ 160.538722] nix-daemon[843]: reaped child process 1598, status = succeeded worker2 # [ 160.632592] nix-grpc-daemon[1623]: time=2026-09-22T16:56:21.279Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 160.737921] nix-daemon[843]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker2 # [ 160.753510] nix-daemon[843]: reaped child process 1638, status = succeeded lb # [ 160.789731] niks3-server[1154]: 2026/09/22 16:56:21 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 160.775253] nix-grpc-daemon[1623]: time=2026-09-22T16:56:21.420Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 160.779291] nix-grpc-daemon[1623]: time=2026-09-22T16:56:21.420Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)" lb # [ 160.843615] niks3-server[1154]: 2026/09/22 16:56:21 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst worker2 # [ 160.826348] nix-grpc-daemon[1623]: time=2026-09-22T16:56:21.472Z level=INFO msg="Uploading 1 narinfos" lb # [ 160.859841] niks3-server[1154]: 2026/09/22 16:56:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign lb # [ 160.870727] niks3-server[1154]: 2026/09/22 16:56:21 INFO Signed narinfos id=52 count=1 lb # [ 160.876347] niks3-server[1154]: 2026/09/22 16:56:21 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv lb # [ 160.889363] niks3-server[1154]: 2026/09/22 16:56:21 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls worker2 # [ 160.866567] nix-grpc-daemon[1623]: time=2026-09-22T16:56:21.513Z level=INFO msg="Upload complete. (230ms)" lb # [ 160.893278] niks3-server[1154]: 2026/09/22 16:56:21 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete lb # [ 160.902134] niks3-server[1154]: 2026/09/22 16:56:21 INFO Completed upload id=52 worker2 # [ 160.876295] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43910 duration_s=5 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=1 lb # [ 160.905427] niks3-server[1154]: 2026/09/22 16:56:21 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo worker2 # [ 160.892212] nix-daemon[843]: reaped child process 1596, status = succeeded client # [ 160.904768] systemd[1]: mid.service: Deactivated successfully. client # [ 160.909094] systemd[1]: mid.service: Consumed 210ms CPU time over 8.454s wall clock time, 41.5M memory peak, 8.6K incoming IP traffic, 10.3K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active mid, in 1.12 seconds) client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; } client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.05 seconds) (finished: subtest: scheduler restart mid-build: build finishes, no double build, in 9.25 seconds) subtest: interrupting the client stops the build on the worker client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr client # [ 161.679273] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr. client # Running as unit: intr.service; invocation ID: 59b5bee707734241bd2a0267e1b84d49 client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.06 seconds) client # [ 161.856770] nix[1482]: don't know how to build these paths: client # [ 161.857753] nix[1482]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker2 # [ 161.867556] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:43910 worker1 # [ 161.935958] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:22Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=0 worker1 # [ 161.936377] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:22Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 161.985396] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 162.001724] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 162.023612] nix-grpc-daemon[1595]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 4 ms (attempt 1/5) worker1 # [ 162.035854] nix-grpc-daemon[1595]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 140 ms (attempt 2/5) worker1 # [ 162.185516] nix-grpc-daemon[1595]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 59 ms (attempt 3/5) worker1 # [ 162.246257] nix-grpc-daemon[1595]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 21 ms (attempt 4/5) worker1 # [ 162.269368] nix-grpc-daemon[1595]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 162.275731] nix-grpc-daemon[1595]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 162.293396] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 client # [ 162.244180] nix[1482]: copying 1 paths to 'lb:50051'... worker1 # [ 162.308820] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 client # [ 162.246471] nix[1482]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'... worker1 # [ 162.324633] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 162.351718] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:22Z level=info event=niks3_push_started pid=1671 worker1 # [ 162.500474] nix-grpc-daemon[1671]: time=2026-09-22T16:56:23.135Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 162.605356] nix-daemon[851]: accepted connection from pid 1683, user nix-grpc-daemon (trusted) worker1 # [ 162.625623] nix-daemon[851]: reaped child process 1686, status = succeeded lb # [ 162.600813] niks3-server[1154]: 2026/09/22 16:56:23 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 162.636080] nix-grpc-daemon[1671]: time=2026-09-22T16:56:23.271Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 162.637534] nix-grpc-daemon[1671]: time=2026-09-22T16:56:23.271Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)" lb # [ 162.647514] niks3-server[1154]: 2026/09/22 16:56:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign worker1 # [ 162.681683] nix-grpc-daemon[1671]: time=2026-09-22T16:56:23.316Z level=INFO msg="Uploading 1 narinfos" lb # [ 162.659604] niks3-server[1154]: 2026/09/22 16:56:23 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst lb # [ 162.670493] niks3-server[1154]: 2026/09/22 16:56:23 INFO Signed narinfos id=53 count=1 worker1 # [ 162.703407] nix-grpc-daemon[1671]: time=2026-09-22T16:56:23.338Z level=INFO msg="Upload complete. (199ms)" lb # [ 162.680291] niks3-server[1154]: 2026/09/22 16:56:23 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls lb # [ 162.685122] niks3-server[1154]: 2026/09/22 16:56:23 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete lb # [ 162.689537] niks3-server[1154]: 2026/09/22 16:56:23 INFO Completed upload id=53 worker1 # [ 162.712213] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 paths=1 nar_bytes_in=424 lb # [ 162.691452] niks3-server[1154]: 2026/09/22 16:56:23 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo worker1 # [ 162.723730] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 162.729573] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 162.733941] nix-daemon[851]: reaped child process 1669, status = succeeded client # [ 162.679178] nix[1482]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051... worker1 # [ 162.746196] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) client: must succeed: systemctl kill -s INT intr worker1 # [ 164.231869] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:24Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv client # [ 164.166688] systemd[1]: intr.service: Sent signal SIGINT to main process 1482 (nix) on client request. client # [ 164.175721] nix[1482]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client # [ 164.188696] nix[1482]: error: interrupted by the user client: (finished: must succeed: systemctl kill -s INT intr, in 0.08 seconds) client # [ 164.213877] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE client # [ 164.218628] systemd[1]: intr.service: Failed with result 'exit-code'. client # [ 164.221500] systemd[1]: intr.service: Consumed 170ms CPU time over 2.537s wall clock time, 41.5M memory peak, 6.4K incoming IP traffic, 8.4K outgoing IP traffic. worker1 # [ 164.368512] nix-daemon[1690]: unexpected Nix daemon error: error: interrupted by the user worker1 # [ 164.377335] nix-daemon[851]: reaped child process 1688, status = succeeded worker1 # [ 164.381685] nix-daemon[851]: reaped child process 1690, status = failed with exit code 1 (finished: subtest: interrupting the client stops the build on the worker, in 3.83 seconds) subtest: a deploy mid-build drains: the build finishes, the new generation takes over client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw client # [ 165.533185] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw. client # Running as unit: sw.service; invocation ID: 9eec3e288831490aa0950ddef08f944c client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.09 seconds) client # [ 165.745203] nix[1503]: don't know how to build these paths: client # [ 165.746295] nix[1503]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker2 # [ 165.757043] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:43910 worker1 # [ 165.822340] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:26Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=0 worker1 # [ 165.827927] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:26Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 165.816432] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43910 worker2 # [ 165.832342] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43910 client # [ 165.834867] nix[1503]: copying 1 paths to 'lb:50051'... worker2 # [ 165.846977] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43910 client # [ 165.837670] nix[1503]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'... worker2 # [ 165.862012] nix-daemon[843]: accepted connection from pid 1521, user nix-grpc-daemon (trusted) worker2 # [ 166.019797] nix-daemon[843]: accepted connection from pid 1666, user nix-grpc-daemon (trusted) worker2 # [ 166.039140] nix-daemon[843]: reaped child process 1669, status = succeeded lb # [ 166.072768] niks3-server[1154]: 2026/09/22 16:56:26 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 166.055182] nix-grpc-daemon[1623]: time=2026-09-22T16:56:26.702Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 166.057747] nix-grpc-daemon[1623]: time=2026-09-22T16:56:26.704Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)" lb # [ 166.104734] niks3-server[1154]: 2026/09/22 16:56:26 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst worker2 # [ 166.088873] nix-grpc-daemon[1623]: time=2026-09-22T16:56:26.734Z level=INFO msg="Uploading 1 narinfos" lb # [ 166.118366] niks3-server[1154]: 2026/09/22 16:56:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign lb # [ 166.130330] niks3-server[1154]: 2026/09/22 16:56:26 INFO Signed narinfos id=54 count=1 worker2 # [ 166.112549] nix-grpc-daemon[1623]: time=2026-09-22T16:56:26.759Z level=INFO msg="Upload complete. (233ms)" lb # [ 166.139926] niks3-server[1154]: 2026/09/22 16:56:26 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls lb # [ 166.145824] niks3-server[1154]: 2026/09/22 16:56:26 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete lb # [ 166.151321] niks3-server[1154]: 2026/09/22 16:56:26 INFO Completed upload id=54 worker2 # [ 166.120171] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43910 duration_s=0 paths=1 nar_bytes_in=416 lb # [ 166.155577] niks3-server[1154]: 2026/09/22 16:56:26 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo worker2 # [ 166.136683] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43910 worker2 # [ 166.145882] nix-daemon[843]: accepted connection from pid 1521, user nix-grpc-daemon (trusted) worker2 # [ 166.151217] nix-daemon[843]: reaped child process 1664, status = succeeded worker2 # [ 166.164157] nix-daemon[843]: accepted connection from pid 1521, user nix-grpc-daemon (trusted) client # [ 166.150677] nix[1503]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051... worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2 worker2 # Checking switch inhibitors... done worker2 # [ 168.546174] nixos[1705]: switching to system configuration /nix/store/v5mnmfs27v4kkcfdrk24lffriqdnfny8-nixos-system-worker2-test worker2 # activating the configuration... worker2 # [ 168.556174] systemd[1]: Stopped target Local File Systems. worker2 # [ 168.557071] systemd[1]: Stopped target Remote File Systems. worker2 # setting up /etc... worker2 # [ 169.109204] systemd[1]: Reload requested from client PID 1705 ('.switch-to-conf') (unit backdoor.service)... worker2 # [ 169.110837] systemd[1]: Reloading... lb # [ 169.149493] envoy[895]: conn peer=[2001:db8:1::1]:55172 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3940 tx=1941 ms=8227 lb # [ 169.154575] envoy[895]: conn peer=[2001:db8:1::1]:35698 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3000 tx=802 ms=2341 worker2 # [ 173.956428] systemd[1]: Reloading finished in 4839 ms. worker2 # restarting sysinit-reactivation.target worker2 # [ 174.088466] systemd[1]: Starting Re-setup tmpfiles on a system that is already running.... worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service worker2 # [ 174.222397] systemd[1]: Finished Re-setup tmpfiles on a system that is already running.. worker2 # [ 174.223628] systemd[1]: Reached target Reactivate sysinit units. worker2 # [ 174.237103] systemd[1]: Reloading D-Bus System Message Bus... worker2 # [ 174.238303] dbus-broker-launch[488]: Caught SIGHUP, trigger reload. worker2 # [ 174.239237] systemd[1]: Reloading Nix worker-protocol over gRPC... worker2 # [ 174.248136] dbus-broker-launch[488]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 174.249417] dbus-broker-launch[488]: NSS returned no entry for 'systemd-timesync' worker2 # [ 174.252762] dbus-broker-launch[488]: Invalid user-name in /nix/store/ng2ydfkv03gvmdws4f4mjf9wj6qj3kq6-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker2 # [ 174.285302] systemd[1]: Reloaded Nix worker-protocol over gRPC. worker2 # [ 174.294231] systemd[1]: Reloaded D-Bus System Message Bus. worker2 # [ 174.298057] systemd[1]: Reached target Remote File Systems. worker2 # [ 174.309291] systemd[1]: Starting linger-users.service... worker2 # [ 174.316784] systemd[1]: Reached target Local File Systems. worker2 # [ 174.317666] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker2 # [ 174.334543] systemd[1]: Starting Create SUID/SGID Wrappers... worker2 # [ 174.335470] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker2 # [ 174.337046] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker2 # [ 174.339198] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 174.353122] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 174.358531] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 174.366547] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 174.370513] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 174.387862] systemd[1]: linger-users.service: Deactivated successfully. worker2 # [ 174.389075] systemd[1]: Finished linger-users.service. worker2 # [ 174.542666] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:35Z level=info event=draining worker2 # [ 174.706921] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 174.712184] systemd[1]: Finished Create SUID/SGID Wrappers. worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service worker2 # [ 174.963374] nixos[1705]: finished switching to system configuration /nix/store/v5mnmfs27v4kkcfdrk24lffriqdnfny8-nixos-system-worker2-test worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 6.87 seconds) worker2: must succeed: pgrep -f 'read -t [9]0 x' worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2 worker2 # [ 175.249743] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:43910 client # don't know how to build these paths: client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker1 # [ 175.313446] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 175.341525] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:35Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=0 worker1 # [ 175.349005] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:35Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 175.390230] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 175.422566] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'... worker1 # [ 175.439619] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 175.457958] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 175.629029] nix-daemon[851]: accepted connection from pid 1745, user nix-grpc-daemon (trusted) worker1 # [ 175.649867] nix-daemon[851]: reaped child process 1748, status = succeeded lb # [ 175.632835] niks3-server[1154]: 2026/09/22 16:56:36 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 175.668367] nix-grpc-daemon[1671]: time=2026-09-22T16:56:36.302Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 175.670011] nix-grpc-daemon[1671]: time=2026-09-22T16:56:36.302Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)" lb # [ 175.668454] niks3-server[1154]: 2026/09/22 16:56:36 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst worker1 # [ 175.706581] nix-grpc-daemon[1671]: time=2026-09-22T16:56:36.341Z level=INFO msg="Uploading 1 narinfos" lb # [ 175.684461] niks3-server[1154]: 2026/09/22 16:56:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign lb # [ 175.693194] niks3-server[1154]: 2026/09/22 16:56:36 INFO Signed narinfos id=55 count=1 lb # [ 175.698885] niks3-server[1154]: 2026/09/22 16:56:36 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls worker1 # [ 175.732950] nix-grpc-daemon[1671]: time=2026-09-22T16:56:36.368Z level=INFO msg="Upload complete. (257ms)" lb # [ 175.709477] niks3-server[1154]: 2026/09/22 16:56:36 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete lb # [ 175.714352] niks3-server[1154]: 2026/09/22 16:56:36 INFO Completed upload id=55 lb # [ 175.717580] niks3-server[1154]: 2026/09/22 16:56:36 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo worker1 # [ 175.742430] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 paths=1 nar_bytes_in=528 worker1 # [ 175.753229] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 175.765576] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 175.770878] nix-daemon[851]: reaped child process 1743, status = succeeded client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051... worker1 # [ 175.782111] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) client # farm-a> node-a: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-during-drain worker1 # [ 176.889419] nix-daemon[851]: reaped child process 1752, status = succeeded worker1 # [ 176.958681] nix-daemon[851]: accepted connection from pid 1768, user nix-grpc-daemon (trusted) worker1 # [ 176.982108] nix-daemon[851]: reaped child process 1771, status = succeeded lb # [ 176.961384] niks3-server[1154]: 2026/09/22 16:56:37 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 176.996406] nix-grpc-daemon[1671]: time=2026-09-22T16:56:37.630Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 176.997998] nix-grpc-daemon[1671]: time=2026-09-22T16:56:37.630Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)" lb # [ 177.019171] niks3-server[1154]: 2026/09/22 16:56:37 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv lb # [ 177.033592] niks3-server[1154]: 2026/09/22 16:56:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign lb # [ 177.047852] niks3-server[1154]: 2026/09/22 16:56:37 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst lb # [ 177.056951] niks3-server[1154]: 2026/09/22 16:56:37 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls lb # [ 177.070367] niks3-server[1154]: 2026/09/22 16:56:37 INFO Signed narinfos id=56 count=1 worker1 # [ 177.095707] nix-grpc-daemon[1671]: time=2026-09-22T16:56:37.730Z level=INFO msg="Uploading 1 narinfos" lb # [ 177.081102] niks3-server[1154]: 2026/09/22 16:56:37 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete lb # [ 177.093673] niks3-server[1154]: 2026/09/22 16:56:37 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo lb # [ 177.100167] niks3-server[1154]: 2026/09/22 16:56:37 INFO Completed upload id=56 worker1 # [ 177.125843] nix-grpc-daemon[1671]: time=2026-09-22T16:56:37.761Z level=INFO msg="Upload complete. (330ms)" worker1 # [ 177.144574] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=1 worker1 # [ 177.164470] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:37Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=0 worker1 # [ 177.169321] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:37Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 177.174046] nix-daemon[851]: reaped child process 1750, status = succeeded worker1 # [ 177.189513] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 177.205160] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 177.221038] nix-grpc-daemon[1671]: time=2026-09-22T16:56:37.856Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker1 # [ 177.232108] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'... worker1 # [ 177.254888] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 177.426373] nix-daemon[851]: accepted connection from pid 1777, user nix-grpc-daemon (trusted) worker1 # [ 177.450445] nix-daemon[851]: reaped child process 1780, status = succeeded lb # [ 177.423611] niks3-server[1154]: 2026/09/22 16:56:38 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 177.460184] nix-grpc-daemon[1671]: time=2026-09-22T16:56:38.095Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 177.462876] nix-grpc-daemon[1671]: time=2026-09-22T16:56:38.097Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)" lb # [ 177.466239] niks3-server[1154]: 2026/09/22 16:56:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign worker1 # [ 177.497042] nix-grpc-daemon[1671]: time=2026-09-22T16:56:38.131Z level=INFO msg="Uploading 1 narinfos" lb # [ 177.480813] niks3-server[1154]: 2026/09/22 16:56:38 INFO Signed narinfos id=57 count=1 lb # [ 177.483528] niks3-server[1154]: 2026/09/22 16:56:38 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst lb # [ 177.494786] niks3-server[1154]: 2026/09/22 16:56:38 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls worker1 # [ 177.527672] nix-grpc-daemon[1671]: time=2026-09-22T16:56:38.162Z level=INFO msg="Upload complete. (254ms)" lb # [ 177.504970] niks3-server[1154]: 2026/09/22 16:56:38 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete lb # [ 177.508890] niks3-server[1154]: 2026/09/22 16:56:38 INFO Completed upload id=57 lb # [ 177.511796] niks3-server[1154]: 2026/09/22 16:56:38 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo worker1 # [ 177.540313] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 paths=1 nar_bytes_in=640 worker1 # [ 177.555724] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 177.562373] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 177.567472] nix-daemon[851]: reaped child process 1775, status = succeeded client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051... worker1 # [ 177.579395] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) client # farm-b> node-a: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-during-drain worker1 # [ 178.418260] nix-daemon[851]: reaped child process 1784, status = succeeded worker1 # [ 178.523480] nix-daemon[851]: accepted connection from pid 1800, user nix-grpc-daemon (trusted) worker1 # [ 178.543301] nix-daemon[851]: reaped child process 1803, status = succeeded lb # [ 178.523955] niks3-server[1154]: 2026/09/22 16:56:39 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 178.559532] nix-grpc-daemon[1671]: time=2026-09-22T16:56:39.193Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 178.563488] nix-grpc-daemon[1671]: time=2026-09-22T16:56:39.194Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)" lb # [ 178.568215] niks3-server[1154]: 2026/09/22 16:56:39 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv lb # [ 178.582183] niks3-server[1154]: 2026/09/22 16:56:39 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst worker1 # [ 178.622156] nix-grpc-daemon[1671]: time=2026-09-22T16:56:39.257Z level=INFO msg="Uploading 1 narinfos" lb # [ 178.600281] niks3-server[1154]: 2026/09/22 16:56:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign lb # [ 178.607946] niks3-server[1154]: 2026/09/22 16:56:39 INFO Signed narinfos id=58 count=1 lb # [ 178.610341] niks3-server[1154]: 2026/09/22 16:56:39 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls worker1 # [ 178.646130] nix-grpc-daemon[1671]: time=2026-09-22T16:56:39.281Z level=INFO msg="Upload complete. (254ms)" lb # [ 178.622208] niks3-server[1154]: 2026/09/22 16:56:39 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete lb # [ 178.627027] niks3-server[1154]: 2026/09/22 16:56:39 INFO Completed upload id=58 lb # [ 178.632540] niks3-server[1154]: 2026/09/22 16:56:39 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo worker1 # [ 178.661187] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=1 worker1 # [ 178.682359] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:39Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=0 worker1 # [ 178.689328] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:39Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 178.694363] nix-daemon[851]: reaped child process 1782, status = succeeded worker1 # [ 178.729728] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 178.746620] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 178.762960] nix-grpc-daemon[1671]: time=2026-09-22T16:56:39.398Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'... worker1 # [ 178.775126] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 178.793504] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 178.955647] nix-daemon[851]: accepted connection from pid 1809, user nix-grpc-daemon (trusted) worker1 # [ 178.978507] nix-daemon[851]: reaped child process 1812, status = succeeded lb # [ 178.957566] niks3-server[1154]: 2026/09/22 16:56:39 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 178.992042] nix-grpc-daemon[1671]: time=2026-09-22T16:56:39.627Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 178.993642] nix-grpc-daemon[1671]: time=2026-09-22T16:56:39.627Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)" lb # [ 178.992436] niks3-server[1154]: 2026/09/22 16:56:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign worker1 # [ 179.023786] nix-grpc-daemon[1671]: time=2026-09-22T16:56:39.658Z level=INFO msg="Uploading 1 narinfos" lb # [ 179.009270] niks3-server[1154]: 2026/09/22 16:56:39 INFO Signed narinfos id=59 count=1 lb # [ 179.015883] niks3-server[1154]: 2026/09/22 16:56:39 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst lb # [ 179.029148] niks3-server[1154]: 2026/09/22 16:56:39 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls worker1 # [ 179.060275] nix-grpc-daemon[1671]: time=2026-09-22T16:56:39.695Z level=INFO msg="Upload complete. (249ms)" lb # [ 179.036377] niks3-server[1154]: 2026/09/22 16:56:39 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete lb # [ 179.042633] niks3-server[1154]: 2026/09/22 16:56:39 INFO Completed upload id=59 worker1 # [ 179.070398] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 paths=1 nar_bytes_in=768 lb # [ 179.048537] niks3-server[1154]: 2026/09/22 16:56:39 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo worker1 # [ 179.088363] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 179.097257] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 179.103346] nix-daemon[851]: reaped child process 1807, status = succeeded client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051... client # farm-top> node-a: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker1 # [ 179.120353] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-during-drain worker1 # [ 179.751776] nix-daemon[851]: reaped child process 1816, status = succeeded worker1 # [ 179.859971] nix-daemon[851]: accepted connection from pid 1832, user nix-grpc-daemon (trusted) worker1 # [ 179.882255] nix-daemon[851]: reaped child process 1835, status = succeeded lb # [ 179.865986] niks3-server[1154]: 2026/09/22 16:56:40 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 179.904467] nix-grpc-daemon[1671]: time=2026-09-22T16:56:40.537Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 179.906084] nix-grpc-daemon[1671]: time=2026-09-22T16:56:40.537Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)" lb # [ 179.911514] niks3-server[1154]: 2026/09/22 16:56:40 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv lb # [ 179.925916] niks3-server[1154]: 2026/09/22 16:56:40 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst worker1 # [ 179.958572] nix-grpc-daemon[1671]: time=2026-09-22T16:56:40.593Z level=INFO msg="Uploading 1 narinfos" lb # [ 179.937506] niks3-server[1154]: 2026/09/22 16:56:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign lb # [ 179.944467] niks3-server[1154]: 2026/09/22 16:56:40 INFO Signed narinfos id=60 count=1 lb # [ 179.946389] niks3-server[1154]: 2026/09/22 16:56:40 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls lb # [ 179.960101] niks3-server[1154]: 2026/09/22 16:56:40 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete lb # [ 179.968239] niks3-server[1154]: 2026/09/22 16:56:40 INFO Completed upload id=60 worker1 # [ 179.990991] nix-grpc-daemon[1671]: time=2026-09-22T16:56:40.626Z level=INFO msg="Upload complete. (259ms)" lb # [ 179.972702] niks3-server[1154]: 2026/09/22 16:56:40 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo worker1 # [ 179.997311] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=1 worker1 # [ 180.008222] nix-daemon[851]: reaped child process 1814, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 4.97 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.19 seconds) worker2: must succeed: pkill -f 'read -t [9]0 x' worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.06 seconds) client: waiting for success: ! systemctl is-active sw worker2 # [ 180.453507] nix-daemon[843]: reaped child process 1673, status = succeeded worker2 # [ 180.590312] nix-daemon[843]: accepted connection from pid 1846, user nix-grpc-daemon (trusted) worker2 # [ 180.617387] nix-daemon[843]: reaped child process 1849, status = succeeded lb # [ 180.648731] niks3-server[1154]: 2026/09/22 16:56:41 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 180.632605] nix-grpc-daemon[1623]: time=2026-09-22T16:56:41.278Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 180.634360] nix-grpc-daemon[1623]: time=2026-09-22T16:56:41.278Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)" lb # [ 180.694463] niks3-server[1154]: 2026/09/22 16:56:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign worker2 # [ 180.673169] nix-grpc-daemon[1623]: time=2026-09-22T16:56:41.319Z level=INFO msg="Uploading 1 narinfos" lb # [ 180.712303] niks3-server[1154]: 2026/09/22 16:56:41 INFO Signed narinfos id=61 count=1 lb # [ 180.714750] niks3-server[1154]: 2026/09/22 16:56:41 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls lb # [ 180.726787] niks3-server[1154]: 2026/09/22 16:56:41 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv lb # [ 180.736944] niks3-server[1154]: 2026/09/22 16:56:41 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst worker2 # [ 180.714879] nix-grpc-daemon[1623]: time=2026-09-22T16:56:41.361Z level=INFO msg="Upload complete. (320ms)" lb # [ 180.743834] niks3-server[1154]: 2026/09/22 16:56:41 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete lb # [ 180.746733] niks3-server[1154]: 2026/09/22 16:56:41 INFO Completed upload id=61 lb # [ 180.751959] niks3-server[1154]: 2026/09/22 16:56:41 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo worker2 # [ 180.729436] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43910 duration_s=14 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=1 worker2 # [ 180.751196] nix-daemon[843]: reaped child process 1671, status = succeeded client # [ 180.752556] systemd[1]: sw.service: Deactivated successfully. client # [ 180.758140] systemd[1]: sw.service: Consumed 207ms CPU time over 15.223s wall clock time, 39.4M memory peak, 7.3K incoming IP traffic, 8.5K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active sw, in 1.11 seconds) client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; } client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.06 seconds) worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 0.04 seconds) lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker2 # [ 181.550437] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:42Z level=info event=signal_exit lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 183.604161] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:44Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:50026 worker1 # [ 183.609073] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:44Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 183.560827] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:44Z level=info event=scheduler_restarting worker2 # [ 183.581122] nix-grpc-daemon[1521]: ts=2026-09-22T16:56:44Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 183.612864] nix-daemon[843]: reaped child process 1577, status = succeeded lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 184.360235] nix-grpc-daemon[1521]: E0000 00:00:1790096205.007085 1566 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true worker2 # [ 184.367595] nix-grpc-daemon[1521]: E0000 00:00:1790096205.007339 1566 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 185.597633] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 185.604189] systemd[1]: nix-grpc-daemon.service: Consumed 849ms CPU time over 34.211s wall clock time, 35.9M memory peak, 21.4K incoming IP traffic, 29.6K outgoing IP traffic. lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 186.595206] systemd[1]: Starting scheduler-token.service... worker2 # [ 186.688172] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 186.696234] systemd[1]: Finished scheduler-token.service. worker2 # [ 186.698484] systemd[1]: scheduler-token.service: Consumed 62ms CPU time over 97ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 186.720704] systemd[1]: Starting Nix worker-protocol over gRPC... lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 186.858541] nix-grpc-daemon[1864]: ts=2026-09-22T16:56:47Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 186.861277] nix-grpc-daemon[1864]: ts=2026-09-22T16:56:47Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 186.865622] nix-grpc-daemon[1864]: ts=2026-09-22T16:56:47Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 186.872105] nix-grpc-daemon[1864]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 186.873765] nix-grpc-daemon[1864]: E0000 00:00:1790096207.517910 1864 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 5.46 seconds) lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 186.924112] nix-grpc-daemon[1864]: E0000 00:00:1790096207.525341 1864 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 186.932437] nix-grpc-daemon[1864]: ts=2026-09-22T16:56:47Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 186.935084] nix-grpc-daemon[1864]: E0000 00:00:1790096207.541074 1904 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 186.941355] nix-grpc-daemon[1864]: E0000 00:00:1790096207.559558 1904 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 186.946721] nix-grpc-daemon[1864]: E0000 00:00:1790096207.559813 1904 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 186.949723] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 186.950582] nix-grpc-daemon[1864]: E0000 00:00:1790096207.578303 1904 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 186.953103] nix-grpc-daemon[1864]: ts=2026-09-22T16:56:47Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 187.002030] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:47Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:50026 worker1 # [ 187.008733] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:47Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 21.56 seconds) subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1 worker1 # [ 187.334092] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:47Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:48662 worker2 # [ 187.295068] nix-grpc-daemon[1864]: ts=2026-09-22T16:56:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:35652 worker2 # [ 187.311491] nix-daemon[843]: accepted connection from pid 1864, user nix-grpc-daemon (trusted) worker2 # [ 187.338051] nix-grpc-daemon[1864]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 90 ms (attempt 1/5) worker2 # [ 187.437454] nix-grpc-daemon[1864]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 48 ms (attempt 2/5) worker2 # [ 187.487599] nix-grpc-daemon[1864]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 184 ms (attempt 3/5) worker2 # [ 187.674022] nix-grpc-daemon[1864]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 170 ms (attempt 4/5) worker2 # [ 187.846772] nix-grpc-daemon[1864]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 187.855726] nix-grpc-daemon[1864]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 187.966117] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:48Z level=debug event=want client=5 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=0 worker1 # [ 187.974184] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:48Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 187.980527] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:48Z level=debug event=want client=5 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=0 worker1 # [ 187.985818] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:48Z level=debug event=want client=5 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=0 worker1 # [ 188.015576] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 188.035868] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 188.054438] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 188.071635] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 188.244570] nix-daemon[851]: accepted connection from pid 1844, user nix-grpc-daemon (trusted) worker1 # [ 188.269855] nix-daemon[851]: reaped child process 1847, status = succeeded lb # [ 188.253851] niks3-server[1154]: 2026/09/22 16:56:48 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 188.292570] nix-grpc-daemon[1671]: time=2026-09-22T16:56:48.926Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 188.294293] nix-grpc-daemon[1671]: time=2026-09-22T16:56:48.926Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)" lb # [ 188.297117] niks3-server[1154]: 2026/09/22 16:56:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign worker1 # [ 188.331104] nix-grpc-daemon[1671]: time=2026-09-22T16:56:48.966Z level=INFO msg="Uploading 1 narinfos" lb # [ 188.310301] niks3-server[1154]: 2026/09/22 16:56:48 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst lb # [ 188.320950] niks3-server[1154]: 2026/09/22 16:56:48 INFO Signed narinfos id=62 count=1 lb # [ 188.330310] niks3-server[1154]: 2026/09/22 16:56:48 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls lb # [ 188.341889] niks3-server[1154]: 2026/09/22 16:56:48 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete worker1 # [ 188.373739] nix-grpc-daemon[1671]: time=2026-09-22T16:56:49.008Z level=INFO msg="Upload complete. (282ms)" lb # [ 188.353497] niks3-server[1154]: 2026/09/22 16:56:49 INFO Completed upload id=62 lb # [ 188.357530] niks3-server[1154]: 2026/09/22 16:56:49 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo worker1 # [ 188.384551] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 188.408791] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 188.416912] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 188.423383] nix-daemon[851]: reaped child process 1842, status = succeeded worker1 # [ 188.436267] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) lb # [ 189.152813] envoy[895]: conn peer=[2001:db8:1::1]:53936 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9591 tx=4401 ms=4722 lb # [ 189.157878] envoy[895]: conn peer=[2001:db8:1::1]:35714 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3004 tx=1518 ms=14987 lb # [ 189.162991] envoy[895]: conn peer=[2001:db8:1::4]:57486 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=2255 tx=442 ms=31973 worker1 # [ 189.283392] nix-daemon[851]: reaped child process 1851, status = succeeded worker1 # [ 189.398864] nix-daemon[851]: accepted connection from pid 1867, user nix-grpc-daemon (trusted) worker1 # [ 189.424783] nix-daemon[851]: reaped child process 1870, status = succeeded lb # [ 189.398209] niks3-server[1154]: 2026/09/22 16:56:50 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 189.434566] nix-grpc-daemon[1671]: time=2026-09-22T16:56:50.067Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 189.437690] nix-grpc-daemon[1671]: time=2026-09-22T16:56:50.068Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)" lb # [ 189.453190] niks3-server[1154]: 2026/09/22 16:56:50 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls worker1 # [ 189.488397] nix-grpc-daemon[1671]: time=2026-09-22T16:56:50.123Z level=INFO msg="Uploading 1 narinfos" lb # [ 189.468100] niks3-server[1154]: 2026/09/22 16:56:50 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv lb # [ 189.471572] niks3-server[1154]: 2026/09/22 16:56:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign lb # [ 189.483170] niks3-server[1154]: 2026/09/22 16:56:50 INFO Signed narinfos id=63 count=1 lb # [ 189.492514] niks3-server[1154]: 2026/09/22 16:56:50 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst worker1 # [ 189.516516] nix-grpc-daemon[1671]: time=2026-09-22T16:56:50.151Z level=INFO msg="Upload complete. (266ms)" lb # [ 189.495635] niks3-server[1154]: 2026/09/22 16:56:50 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete lb # [ 189.502885] niks3-server[1154]: 2026/09/22 16:56:50 INFO Completed upload id=63 lb # [ 189.505713] niks3-server[1154]: 2026/09/22 16:56:50 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo worker1 # [ 189.531206] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:50Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 189.546933] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=7 outputs=1 worker1 # [ 189.552882] nix-daemon[851]: reaped child process 1849, status = succeeded worker1 # [ 189.575976] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 189.586673] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:50Z level=debug event=want client=5 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=0 worker1 # [ 189.598464] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 189.615358] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 189.629575] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 189.805349] nix-daemon[851]: accepted connection from pid 1876, user nix-grpc-daemon (trusted) lb # [ 189.803688] niks3-server[1154]: 2026/09/22 16:56:50 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 189.832120] nix-daemon[851]: reaped child process 1879, status = succeeded worker1 # [ 189.838439] nix-grpc-daemon[1671]: time=2026-09-22T16:56:50.473Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 189.840278] nix-grpc-daemon[1671]: time=2026-09-22T16:56:50.473Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)" worker1 # [ 189.876000] nix-grpc-daemon[1671]: time=2026-09-22T16:56:50.511Z level=INFO msg="Uploading 1 narinfos" lb # [ 189.841428] niks3-server[1154]: 2026/09/22 16:56:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign lb # [ 189.860346] niks3-server[1154]: 2026/09/22 16:56:50 INFO Signed narinfos id=64 count=1 lb # [ 189.867821] niks3-server[1154]: 2026/09/22 16:56:50 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst lb # [ 189.884844] niks3-server[1154]: 2026/09/22 16:56:50 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls worker1 # [ 189.915008] nix-grpc-daemon[1671]: time=2026-09-22T16:56:50.550Z level=INFO msg="Upload complete. (269ms)" lb # [ 189.894102] niks3-server[1154]: 2026/09/22 16:56:50 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete lb # [ 189.899260] niks3-server[1154]: 2026/09/22 16:56:50 INFO Completed upload id=64 lb # [ 189.902938] niks3-server[1154]: 2026/09/22 16:56:50 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo worker1 # [ 189.924420] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 189.944158] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 189.951057] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 189.957126] nix-daemon[851]: reaped child process 1874, status = succeeded worker1 # [ 189.972506] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 190.805846] nix-daemon[851]: reaped child process 1883, status = succeeded worker1 # [ 190.926170] nix-daemon[851]: accepted connection from pid 1899, user nix-grpc-daemon (trusted) worker1 # [ 190.953122] nix-daemon[851]: reaped child process 1902, status = succeeded lb # [ 190.935407] niks3-server[1154]: 2026/09/22 16:56:51 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 191.012668] nix-grpc-daemon[1671]: time=2026-09-22T16:56:51.645Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 191.014950] nix-grpc-daemon[1671]: time=2026-09-22T16:56:51.646Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)" lb # [ 191.029095] niks3-server[1154]: 2026/09/22 16:56:51 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls worker1 # [ 191.065302] nix-grpc-daemon[1671]: time=2026-09-22T16:56:51.700Z level=INFO msg="Uploading 1 narinfos" lb # [ 191.041272] niks3-server[1154]: 2026/09/22 16:56:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign lb # [ 191.051112] niks3-server[1154]: 2026/09/22 16:56:51 INFO Signed narinfos id=65 count=1 lb # [ 191.055024] niks3-server[1154]: 2026/09/22 16:56:51 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv lb # [ 191.065746] niks3-server[1154]: 2026/09/22 16:56:51 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst lb # [ 191.076465] niks3-server[1154]: 2026/09/22 16:56:51 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete worker1 # [ 191.106116] nix-grpc-daemon[1671]: time=2026-09-22T16:56:51.741Z level=INFO msg="Upload complete. (321ms)" lb # [ 191.084595] niks3-server[1154]: 2026/09/22 16:56:51 INFO Completed upload id=65 lb # [ 191.086709] niks3-server[1154]: 2026/09/22 16:56:51 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo worker1 # [ 191.126517] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=8 outputs=1 worker1 # [ 191.143989] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:51Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 191.147650] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:51Z level=debug event=want client=5 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=0 worker1 # [ 191.152718] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 191.157512] nix-daemon[851]: reaped child process 1881, status = succeeded worker1 # [ 191.161736] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 191.176244] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 191.188460] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 191.340229] nix-daemon[851]: accepted connection from pid 1908, user nix-grpc-daemon (trusted) worker1 # [ 191.363248] nix-daemon[851]: reaped child process 1911, status = succeeded lb # [ 191.343974] niks3-server[1154]: 2026/09/22 16:56:51 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 191.378436] nix-grpc-daemon[1671]: time=2026-09-22T16:56:52.013Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 191.380195] nix-grpc-daemon[1671]: time=2026-09-22T16:56:52.013Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)" lb # [ 191.380675] niks3-server[1154]: 2026/09/22 16:56:52 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst worker1 # [ 191.420887] nix-grpc-daemon[1671]: time=2026-09-22T16:56:52.055Z level=INFO msg="Uploading 1 narinfos" lb # [ 191.398487] niks3-server[1154]: 2026/09/22 16:56:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign lb # [ 191.406848] niks3-server[1154]: 2026/09/22 16:56:52 INFO Signed narinfos id=66 count=1 lb # [ 191.413312] niks3-server[1154]: 2026/09/22 16:56:52 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls lb # [ 191.426052] niks3-server[1154]: 2026/09/22 16:56:52 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete lb # [ 191.436767] niks3-server[1154]: 2026/09/22 16:56:52 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo lb # [ 191.441505] niks3-server[1154]: 2026/09/22 16:56:52 INFO Completed upload id=66 worker1 # [ 191.465967] nix-grpc-daemon[1671]: time=2026-09-22T16:56:52.101Z level=INFO msg="Upload complete. (262ms)" worker1 # [ 191.477747] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 191.493004] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 191.507171] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 191.518895] nix-daemon[851]: reaped child process 1906, status = succeeded worker1 # [ 191.528705] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 192.653210] nix-daemon[851]: reaped child process 1915, status = succeeded worker1 # [ 192.726506] nix-daemon[851]: accepted connection from pid 1931, user nix-grpc-daemon (trusted) worker1 # [ 192.756083] nix-daemon[851]: reaped child process 1934, status = succeeded lb # [ 192.729181] niks3-server[1154]: 2026/09/22 16:56:53 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 192.766573] nix-grpc-daemon[1671]: time=2026-09-22T16:56:53.400Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 192.768445] nix-grpc-daemon[1671]: time=2026-09-22T16:56:53.400Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)" lb # [ 192.778786] niks3-server[1154]: 2026/09/22 16:56:53 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker1 # [ 192.815459] nix-grpc-daemon[1671]: time=2026-09-22T16:56:53.450Z level=INFO msg="Uploading 1 narinfos" lb # [ 192.791564] niks3-server[1154]: 2026/09/22 16:56:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign lb # [ 192.801902] niks3-server[1154]: 2026/09/22 16:56:53 INFO Signed narinfos id=67 count=1 lb # [ 192.806984] niks3-server[1154]: 2026/09/22 16:56:53 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst lb # [ 192.816578] niks3-server[1154]: 2026/09/22 16:56:53 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls lb # [ 192.827760] niks3-server[1154]: 2026/09/22 16:56:53 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete lb # [ 192.835115] niks3-server[1154]: 2026/09/22 16:56:53 INFO Completed upload id=67 worker1 # [ 192.857939] nix-grpc-daemon[1671]: time=2026-09-22T16:56:53.493Z level=INFO msg="Upload complete. (313ms)" lb # [ 192.838952] niks3-server[1154]: 2026/09/22 16:56:53 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo worker1 # [ 192.881243] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=9 outputs=1 worker1 # [ 192.898549] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:53Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 192.912494] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 192.921600] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:53Z level=debug event=want client=5 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=0 worker1 # [ 192.928688] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 192.942060] nix-daemon[851]: reaped child process 1913, status = succeeded worker1 # [ 192.948321] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 192.950765] nix-grpc-daemon[1671]: time=2026-09-22T16:56:53.561Z level=INFO msg="All 1 paths already cached" worker1 # [ 192.958694] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 193.096555] nix-daemon[851]: accepted connection from pid 1940, user nix-grpc-daemon (trusted) worker1 # [ 193.120186] nix-daemon[851]: reaped child process 1943, status = succeeded lb # [ 193.101200] niks3-server[1154]: 2026/09/22 16:56:53 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 193.138931] nix-grpc-daemon[1671]: time=2026-09-22T16:56:53.773Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 193.140756] nix-grpc-daemon[1671]: time=2026-09-22T16:56:53.773Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)" lb # [ 193.145003] niks3-server[1154]: 2026/09/22 16:56:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign worker1 # [ 193.177728] nix-grpc-daemon[1671]: time=2026-09-22T16:56:53.812Z level=INFO msg="Uploading 1 narinfos" lb # [ 193.159525] niks3-server[1154]: 2026/09/22 16:56:53 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst lb # [ 193.170884] niks3-server[1154]: 2026/09/22 16:56:53 INFO Signed narinfos id=68 count=1 lb # [ 193.179211] niks3-server[1154]: 2026/09/22 16:56:53 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls worker1 # [ 193.210947] nix-grpc-daemon[1671]: time=2026-09-22T16:56:53.846Z level=INFO msg="Upload complete. (247ms)" lb # [ 193.186980] niks3-server[1154]: 2026/09/22 16:56:53 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete lb # [ 193.194529] niks3-server[1154]: 2026/09/22 16:56:53 INFO Completed upload id=68 worker1 # [ 193.220442] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 193.201204] niks3-server[1154]: 2026/09/22 16:56:53 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo worker1 # [ 193.232888] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 193.246179] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 193.256578] nix-daemon[851]: reaped child process 1938, status = succeeded worker1 # [ 193.265431] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 194.148994] nix-daemon[851]: reaped child process 1947, status = succeeded worker1 # [ 194.266272] nix-daemon[851]: accepted connection from pid 1963, user nix-grpc-daemon (trusted) lb # [ 194.268859] niks3-server[1154]: 2026/09/22 16:56:54 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 194.295675] nix-daemon[851]: reaped child process 1966, status = succeeded worker1 # [ 194.308559] nix-grpc-daemon[1671]: time=2026-09-22T16:56:54.941Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 194.310193] nix-grpc-daemon[1671]: time=2026-09-22T16:56:54.941Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)" lb # [ 194.306990] niks3-server[1154]: 2026/09/22 16:56:54 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv lb # [ 194.315690] niks3-server[1154]: 2026/09/22 16:56:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign worker1 # [ 194.343361] nix-grpc-daemon[1671]: time=2026-09-22T16:56:54.978Z level=INFO msg="Uploading 1 narinfos" lb # [ 194.320619] niks3-server[1154]: 2026/09/22 16:56:54 INFO Signed narinfos id=69 count=1 lb # [ 194.330521] niks3-server[1154]: 2026/09/22 16:56:54 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls lb # [ 194.335548] niks3-server[1154]: 2026/09/22 16:56:54 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst lb # [ 194.342564] niks3-server[1154]: 2026/09/22 16:56:54 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete worker1 # [ 194.371234] nix-grpc-daemon[1671]: time=2026-09-22T16:56:55.006Z level=INFO msg="Upload complete. (292ms)" lb # [ 194.347331] niks3-server[1154]: 2026/09/22 16:56:54 INFO Completed upload id=69 lb # [ 194.351896] niks3-server[1154]: 2026/09/22 16:56:55 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo worker1 # [ 194.377022] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=10 outputs=1 worker1 # [ 194.379721] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:55Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 194.389998] nix-daemon[851]: reaped child process 1945, status = succeeded worker1 # [ 194.422198] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 194.429847] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:55Z level=debug event=want client=5 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=0 worker1 # [ 194.438907] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 194.450921] nix-grpc-daemon[1671]: time=2026-09-22T16:56:55.086Z level=INFO msg="All 1 paths already cached" worker1 # [ 194.459311] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 194.470791] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 194.609455] nix-daemon[851]: accepted connection from pid 1972, user nix-grpc-daemon (trusted) worker1 # [ 194.633795] nix-daemon[851]: reaped child process 1975, status = succeeded lb # [ 194.616528] niks3-server[1154]: 2026/09/22 16:56:55 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 194.653423] nix-grpc-daemon[1671]: time=2026-09-22T16:56:55.288Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 194.655411] nix-grpc-daemon[1671]: time=2026-09-22T16:56:55.288Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)" lb # [ 194.654471] niks3-server[1154]: 2026/09/22 16:56:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign worker1 # [ 194.689109] nix-grpc-daemon[1671]: time=2026-09-22T16:56:55.324Z level=INFO msg="Uploading 1 narinfos" lb # [ 194.671146] niks3-server[1154]: 2026/09/22 16:56:55 INFO Signed narinfos id=70 count=1 lb # [ 194.678036] niks3-server[1154]: 2026/09/22 16:56:55 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst lb # [ 194.690954] niks3-server[1154]: 2026/09/22 16:56:55 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls worker1 # [ 194.724090] nix-grpc-daemon[1671]: time=2026-09-22T16:56:55.359Z level=INFO msg="Upload complete. (241ms)" lb # [ 194.700407] niks3-server[1154]: 2026/09/22 16:56:55 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete lb # [ 194.708601] niks3-server[1154]: 2026/09/22 16:56:55 INFO Completed upload id=70 worker1 # [ 194.732230] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 194.714475] niks3-server[1154]: 2026/09/22 16:56:55 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo worker1 # [ 194.744704] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 194.756669] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 194.762206] nix-daemon[851]: reaped child process 1970, status = succeeded worker1 # [ 194.774918] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 196.009956] nix-daemon[851]: reaped child process 1979, status = succeeded worker1 # [ 196.122505] nix-daemon[851]: accepted connection from pid 1995, user nix-grpc-daemon (trusted) worker1 # [ 196.151655] nix-daemon[851]: reaped child process 1998, status = succeeded lb # [ 196.125982] niks3-server[1154]: 2026/09/22 16:56:56 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 196.166757] nix-grpc-daemon[1671]: time=2026-09-22T16:56:56.800Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 196.168768] nix-grpc-daemon[1671]: time=2026-09-22T16:56:56.800Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)" lb # [ 196.182955] niks3-server[1154]: 2026/09/22 16:56:56 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker1 # [ 196.217806] nix-grpc-daemon[1671]: time=2026-09-22T16:56:56.852Z level=INFO msg="Uploading 1 narinfos" lb # [ 196.197176] niks3-server[1154]: 2026/09/22 16:56:56 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls lb # [ 196.207998] niks3-server[1154]: 2026/09/22 16:56:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign lb # [ 196.211838] niks3-server[1154]: 2026/09/22 16:56:56 INFO Signed narinfos id=71 count=1 worker1 # [ 196.250309] nix-grpc-daemon[1671]: time=2026-09-22T16:56:56.885Z level=INFO msg="Upload complete. (261ms)" lb # [ 196.226072] niks3-server[1154]: 2026/09/22 16:56:56 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst lb # [ 196.235714] niks3-server[1154]: 2026/09/22 16:56:56 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete lb # [ 196.242465] niks3-server[1154]: 2026/09/22 16:56:56 INFO Completed upload id=71 lb # [ 196.246624] niks3-server[1154]: 2026/09/22 16:56:56 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo worker1 # [ 196.268712] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=11 outputs=1 worker1 # [ 196.288376] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:56Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 196.292865] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:56Z level=debug event=want client=5 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=0 worker1 # [ 196.298370] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 196.305346] nix-daemon[851]: reaped child process 1977, status = succeeded worker1 # [ 196.309804] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 196.321043] nix-grpc-daemon[1671]: time=2026-09-22T16:56:56.956Z level=INFO msg="All 1 paths already cached" worker1 # [ 196.330552] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 196.341389] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 196.489454] nix-daemon[851]: accepted connection from pid 2004, user nix-grpc-daemon (trusted) worker1 # [ 196.513618] nix-daemon[851]: reaped child process 2007, status = succeeded lb # [ 196.495077] niks3-server[1154]: 2026/09/22 16:56:57 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 196.529650] nix-grpc-daemon[1671]: time=2026-09-22T16:56:57.164Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 196.531283] nix-grpc-daemon[1671]: time=2026-09-22T16:56:57.164Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)" lb # [ 196.530651] niks3-server[1154]: 2026/09/22 16:56:57 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls worker1 # [ 196.565716] nix-grpc-daemon[1671]: time=2026-09-22T16:56:57.200Z level=INFO msg="Uploading 1 narinfos" lb # [ 196.544523] niks3-server[1154]: 2026/09/22 16:56:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign lb # [ 196.555339] niks3-server[1154]: 2026/09/22 16:56:57 INFO Signed narinfos id=72 count=1 lb # [ 196.562093] niks3-server[1154]: 2026/09/22 16:56:57 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst lb # [ 196.573148] niks3-server[1154]: 2026/09/22 16:56:57 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete lb # [ 196.581646] niks3-server[1154]: 2026/09/22 16:56:57 INFO Completed upload id=72 lb # [ 196.583780] niks3-server[1154]: 2026/09/22 16:56:57 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo worker1 # [ 196.604548] nix-grpc-daemon[1671]: time=2026-09-22T16:56:57.239Z level=INFO msg="Upload complete. (249ms)" worker1 # [ 196.617282] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 196.632652] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 196.640873] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 196.651817] nix-daemon[851]: reaped child process 2002, status = succeeded worker1 # [ 196.662528] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 197.902578] nix-daemon[851]: reaped child process 2011, status = succeeded worker1 # [ 198.019360] nix-daemon[851]: accepted connection from pid 2027, user nix-grpc-daemon (trusted) worker1 # [ 198.045101] nix-daemon[851]: reaped child process 2030, status = succeeded lb # [ 198.026982] niks3-server[1154]: 2026/09/22 16:56:58 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 198.066639] nix-grpc-daemon[1671]: time=2026-09-22T16:56:58.701Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 198.071487] nix-grpc-daemon[1671]: time=2026-09-22T16:56:58.703Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)" lb # [ 198.078934] niks3-server[1154]: 2026/09/22 16:56:58 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst worker1 # [ 198.118535] nix-grpc-daemon[1671]: time=2026-09-22T16:56:58.753Z level=INFO msg="Uploading 1 narinfos" lb # [ 198.094367] niks3-server[1154]: 2026/09/22 16:56:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign lb # [ 198.103430] niks3-server[1154]: 2026/09/22 16:56:58 INFO Signed narinfos id=73 count=1 lb # [ 198.110330] niks3-server[1154]: 2026/09/22 16:56:58 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv lb # [ 198.117569] niks3-server[1154]: 2026/09/22 16:56:58 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls lb # [ 198.123757] niks3-server[1154]: 2026/09/22 16:56:58 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete lb # [ 198.130347] niks3-server[1154]: 2026/09/22 16:56:58 INFO Completed upload id=73 worker1 # [ 198.155427] nix-grpc-daemon[1671]: time=2026-09-22T16:56:58.790Z level=INFO msg="Upload complete. (273ms)" lb # [ 198.135570] niks3-server[1154]: 2026/09/22 16:56:58 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo worker1 # [ 198.177272] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=12 outputs=1 worker1 # [ 198.192383] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:58Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 198.204657] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 198.212413] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:58Z level=debug event=want client=5 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=0 worker1 # [ 198.219877] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 198.226050] nix-daemon[851]: reaped child process 2009, status = succeeded worker1 # [ 198.228453] nix-grpc-daemon[1671]: time=2026-09-22T16:56:58.852Z level=INFO msg="All 2 paths already cached" worker1 # [ 198.238614] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 198.247082] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 198.387768] nix-daemon[851]: accepted connection from pid 2036, user nix-grpc-daemon (trusted) worker1 # [ 198.414828] nix-daemon[851]: reaped child process 2039, status = succeeded lb # [ 198.391420] niks3-server[1154]: 2026/09/22 16:56:59 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 198.425713] nix-grpc-daemon[1671]: time=2026-09-22T16:56:59.060Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 198.427343] nix-grpc-daemon[1671]: time=2026-09-22T16:56:59.060Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)" lb # [ 198.424925] niks3-server[1154]: 2026/09/22 16:56:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign worker1 # [ 198.457566] nix-grpc-daemon[1671]: time=2026-09-22T16:56:59.092Z level=INFO msg="Uploading 1 narinfos" lb # [ 198.439471] niks3-server[1154]: 2026/09/22 16:56:59 INFO Signed narinfos id=74 count=1 lb # [ 198.447453] niks3-server[1154]: 2026/09/22 16:56:59 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst lb # [ 198.461500] niks3-server[1154]: 2026/09/22 16:56:59 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls worker1 # [ 198.493523] nix-grpc-daemon[1671]: time=2026-09-22T16:56:59.128Z level=INFO msg="Upload complete. (238ms)" lb # [ 198.469998] niks3-server[1154]: 2026/09/22 16:56:59 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete worker1 # [ 198.501016] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 198.480289] niks3-server[1154]: 2026/09/22 16:56:59 INFO Completed upload id=74 lb # [ 198.482928] niks3-server[1154]: 2026/09/22 16:56:59 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo worker1 # [ 198.512959] nix-grpc-daemon[1595]: ts=2026-09-22T16:56:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 198.522902] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 198.536492] nix-daemon[851]: reaped child process 2034, status = succeeded worker1 # [ 198.546409] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 199.184650] nix-daemon[851]: reaped child process 2043, status = succeeded worker1 # [ 199.295990] nix-daemon[851]: accepted connection from pid 2059, user nix-grpc-daemon (trusted) worker1 # [ 199.324369] nix-daemon[851]: reaped child process 2062, status = succeeded lb # [ 199.300890] niks3-server[1154]: 2026/09/22 16:56:59 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 199.335539] nix-grpc-daemon[1671]: time=2026-09-22T16:56:59.969Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 199.337412] nix-grpc-daemon[1671]: time=2026-09-22T16:56:59.969Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)" lb # [ 199.342417] niks3-server[1154]: 2026/09/22 16:56:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign worker1 # [ 199.372668] nix-grpc-daemon[1671]: time=2026-09-22T16:57:00.007Z level=INFO msg="Uploading 1 narinfos" lb # [ 199.358445] niks3-server[1154]: 2026/09/22 16:56:59 INFO Signed narinfos id=75 count=1 lb # [ 199.365792] niks3-server[1154]: 2026/09/22 16:57:00 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv lb # [ 199.375808] niks3-server[1154]: 2026/09/22 16:57:00 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls worker1 # [ 199.410544] nix-grpc-daemon[1671]: time=2026-09-22T16:57:00.045Z level=INFO msg="Upload complete. (253ms)" lb # [ 199.387585] niks3-server[1154]: 2026/09/22 16:57:00 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst lb # [ 199.397227] niks3-server[1154]: 2026/09/22 16:57:00 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete lb # [ 199.400123] niks3-server[1154]: 2026/09/22 16:57:00 INFO Completed upload id=75 worker1 # [ 199.422101] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=13 outputs=1 lb # [ 199.402255] niks3-server[1154]: 2026/09/22 16:57:00 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo worker1 # [ 199.438642] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:00Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 199.443353] nix-daemon[851]: reaped child process 2041, status = succeeded worker1 # [ 199.465403] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 199.481848] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 199.498191] nix-grpc-daemon[1671]: time=2026-09-22T16:57:00.133Z level=INFO msg="All 2 paths already cached" worker1 # [ 199.506719] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 199.513380] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 199.648419] nix-daemon[851]: accepted connection from pid 2066, user nix-grpc-daemon (trusted) worker1 # [ 199.674802] nix-daemon[851]: reaped child process 2069, status = succeeded lb # [ 199.647358] niks3-server[1154]: 2026/09/22 16:57:00 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 199.685669] nix-grpc-daemon[1671]: time=2026-09-22T16:57:00.320Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 199.687257] nix-grpc-daemon[1671]: time=2026-09-22T16:57:00.320Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)" lb # [ 199.693050] niks3-server[1154]: 2026/09/22 16:57:00 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls worker1 # [ 199.725289] nix-grpc-daemon[1671]: time=2026-09-22T16:57:00.360Z level=INFO msg="Uploading 1 narinfos" lb # [ 199.707227] niks3-server[1154]: 2026/09/22 16:57:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign lb # [ 199.716361] niks3-server[1154]: 2026/09/22 16:57:00 INFO Signed narinfos id=76 count=1 lb # [ 199.719429] niks3-server[1154]: 2026/09/22 16:57:00 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst worker1 # [ 199.758059] nix-grpc-daemon[1671]: time=2026-09-22T16:57:00.393Z level=INFO msg="Upload complete. (236ms)" lb # [ 199.733994] niks3-server[1154]: 2026/09/22 16:57:00 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete lb # [ 199.739905] niks3-server[1154]: 2026/09/22 16:57:00 INFO Completed upload id=76 worker1 # [ 199.764399] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 199.743035] niks3-server[1154]: 2026/09/22 16:57:00 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo worker1 # [ 199.769446] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 199.778929] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 199.782930] nix-daemon[851]: reaped child process 2064, status = succeeded worker1 # [ 199.792396] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 200.700992] nix-daemon[851]: reaped child process 2073, status = succeeded worker1 # [ 200.771818] nix-daemon[851]: accepted connection from pid 2089, user nix-grpc-daemon (trusted) worker1 # [ 200.798398] nix-daemon[851]: reaped child process 2092, status = succeeded lb # [ 200.782582] niks3-server[1154]: 2026/09/22 16:57:01 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 200.820402] nix-grpc-daemon[1671]: time=2026-09-22T16:57:01.455Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 200.824693] nix-grpc-daemon[1671]: time=2026-09-22T16:57:01.457Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)" lb # [ 200.832901] niks3-server[1154]: 2026/09/22 16:57:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign worker1 # [ 200.864797] nix-grpc-daemon[1671]: time=2026-09-22T16:57:01.499Z level=INFO msg="Uploading 1 narinfos" lb # [ 200.841104] niks3-server[1154]: 2026/09/22 16:57:01 INFO Signed narinfos id=77 count=1 lb # [ 200.850376] niks3-server[1154]: 2026/09/22 16:57:01 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv lb # [ 200.867520] niks3-server[1154]: 2026/09/22 16:57:01 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst lb # [ 200.874105] niks3-server[1154]: 2026/09/22 16:57:01 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls lb # [ 200.883761] niks3-server[1154]: 2026/09/22 16:57:01 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete worker1 # [ 200.915406] nix-grpc-daemon[1671]: time=2026-09-22T16:57:01.550Z level=INFO msg="Upload complete. (311ms)" lb # [ 200.893749] niks3-server[1154]: 2026/09/22 16:57:01 INFO Completed upload id=77 lb # [ 200.896334] niks3-server[1154]: 2026/09/22 16:57:01 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo worker1 # [ 200.925423] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=14 outputs=1 worker1 # [ 200.947777] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:01Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 200.953005] nix-daemon[851]: reaped child process 2071, status = succeeded worker1 # [ 200.979305] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 200.999687] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 201.018712] nix-grpc-daemon[1671]: time=2026-09-22T16:57:01.653Z level=INFO msg="All 2 paths already cached" worker1 # [ 201.031829] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 201.048078] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 201.215256] nix-daemon[851]: accepted connection from pid 2096, user nix-grpc-daemon (trusted) worker1 # [ 201.242571] nix-daemon[851]: reaped child process 2099, status = succeeded lb # [ 201.221626] niks3-server[1154]: 2026/09/22 16:57:01 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 201.256176] nix-grpc-daemon[1671]: time=2026-09-22T16:57:01.891Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 201.258804] nix-grpc-daemon[1671]: time=2026-09-22T16:57:01.893Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)" lb # [ 201.259520] niks3-server[1154]: 2026/09/22 16:57:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign worker1 # [ 201.292709] nix-grpc-daemon[1671]: time=2026-09-22T16:57:01.927Z level=INFO msg="Uploading 1 narinfos" lb # [ 201.275220] niks3-server[1154]: 2026/09/22 16:57:01 INFO Signed narinfos id=78 count=1 lb # [ 201.279547] niks3-server[1154]: 2026/09/22 16:57:01 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst lb # [ 201.295450] niks3-server[1154]: 2026/09/22 16:57:01 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls lb # [ 201.305624] niks3-server[1154]: 2026/09/22 16:57:01 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete worker1 # [ 201.335655] nix-grpc-daemon[1671]: time=2026-09-22T16:57:01.970Z level=INFO msg="Upload complete. (269ms)" lb # [ 201.314071] niks3-server[1154]: 2026/09/22 16:57:01 INFO Completed upload id=78 lb # [ 201.318773] niks3-server[1154]: 2026/09/22 16:57:01 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo worker1 # [ 201.348417] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 201.364424] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 worker1 # [ 201.374369] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 201.380794] nix-daemon[851]: reaped child process 2094, status = succeeded worker1 # [ 201.397148] nix-daemon[851]: accepted connection from pid 1595, user nix-grpc-daemon (trusted) worker1 # [ 202.681305] nix-daemon[851]: reaped child process 2103, status = succeeded worker1 # [ 202.791883] nix-daemon[851]: accepted connection from pid 2119, user nix-grpc-daemon (trusted) worker1 # [ 202.814669] nix-daemon[851]: reaped child process 2122, status = succeeded lb # [ 202.797579] niks3-server[1154]: 2026/09/22 16:57:03 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 202.835814] nix-grpc-daemon[1671]: time=2026-09-22T16:57:03.469Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 202.837749] nix-grpc-daemon[1671]: time=2026-09-22T16:57:03.469Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)" lb # [ 202.846220] niks3-server[1154]: 2026/09/22 16:57:03 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst worker1 # [ 202.885543] nix-grpc-daemon[1671]: time=2026-09-22T16:57:03.520Z level=INFO msg="Uploading 1 narinfos" lb # [ 202.861612] niks3-server[1154]: 2026/09/22 16:57:03 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls lb # [ 202.871090] niks3-server[1154]: 2026/09/22 16:57:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign lb # [ 202.878825] niks3-server[1154]: 2026/09/22 16:57:03 INFO Signed narinfos id=79 count=1 lb # [ 202.886313] niks3-server[1154]: 2026/09/22 16:57:03 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker1 # [ 202.911012] nix-grpc-daemon[1671]: time=2026-09-22T16:57:03.546Z level=INFO msg="Upload complete. (300ms)" lb # [ 202.895288] niks3-server[1154]: 2026/09/22 16:57:03 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete lb # [ 202.900354] niks3-server[1154]: 2026/09/22 16:57:03 INFO Completed upload id=79 lb # [ 202.902533] niks3-server[1154]: 2026/09/22 16:57:03 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo worker1 # [ 202.926545] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48662 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=15 outputs=1 worker1 # [ 202.948795] nix-daemon[851]: reaped child process 2101, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 15.91 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]' client # [ 203.075544] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]". client # Running as unit: gpu.service; invocation ID: d9360422809f409595c6f531f58a6aaf client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1 # [ 203.340369] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:48662 client # [ 203.291176] nix[1583]: don't know how to build these paths: client # [ 203.293455] nix[1583]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv worker2 # [ 203.309489] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:35652 worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) worker1 # [ 203.388434] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:04Z level=debug event=want client=6 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=0 client # [ 203.331790] nix[1583]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip}, waiting up to 30s for a worker lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) client: waiting for success: systemctl show -p Result gpu | grep -q exit-code lb # [ 209.152787] envoy[895]: conn peer=[2001:db8:1::1]:36064 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27071 tx=12428 ms=15612 client # [ 233.333341] nix[1583]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip} client # [ 233.346195] nix[1583]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed client # [ 233.390755] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE client # [ 233.399086] systemd[1]: gpu.service: Failed with result 'exit-code'. client # [ 233.403247] systemd[1]: gpu.service: Consumed 202ms CPU time over 30.320s wall clock time, 35.4M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic. client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.22 seconds) client: must succeed: journalctl -u gpu -o cat client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 46.90 seconds) subtest: low disk drains a worker and builds go to the other worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.17 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free' worker1 # [ 235.667969] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:36Z level=info event=unhealthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.10 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2 client # don't know how to build these paths: client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker2 # [ 236.433150] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:37Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:35652 worker2 # [ 236.442089] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:35652 worker1 # [ 236.516934] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:37Z level=debug event=want client=7 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=0 worker1 # [ 236.517374] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:37Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 236.513398] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35652 worker2 # [ 236.530418] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:35652 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'... worker2 # [ 236.547553] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35652 worker2 # [ 236.563890] nix-daemon[843]: accepted connection from pid 1864, user nix-grpc-daemon (trusted) worker2 # [ 236.594034] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:37Z level=info event=niks3_push_started pid=1924 worker2 # [ 236.743486] nix-grpc-daemon[1924]: time=2026-09-22T16:57:37.390Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 236.855924] nix-daemon[843]: accepted connection from pid 1936, user nix-grpc-daemon (trusted) worker2 # [ 236.879471] nix-daemon[843]: reaped child process 1939, status = succeeded lb # [ 236.906023] niks3-server[1154]: 2026/09/22 16:57:37 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 236.891841] nix-grpc-daemon[1924]: time=2026-09-22T16:57:37.538Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 236.896844] nix-grpc-daemon[1924]: time=2026-09-22T16:57:37.538Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)" lb # [ 236.954175] niks3-server[1154]: 2026/09/22 16:57:37 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst worker2 # [ 236.938394] nix-grpc-daemon[1924]: time=2026-09-22T16:57:37.584Z level=INFO msg="Uploading 1 narinfos" lb # [ 236.970281] niks3-server[1154]: 2026/09/22 16:57:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign lb # [ 236.979130] niks3-server[1154]: 2026/09/22 16:57:37 INFO Signed narinfos id=80 count=1 worker2 # [ 236.968525] nix-grpc-daemon[1924]: time=2026-09-22T16:57:37.615Z level=INFO msg="Upload complete. (221ms)" lb # [ 236.990218] niks3-server[1154]: 2026/09/22 16:57:37 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete lb # [ 237.001917] niks3-server[1154]: 2026/09/22 16:57:37 INFO Completed upload id=80 worker2 # [ 236.976338] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35652 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 237.008711] niks3-server[1154]: 2026/09/22 16:57:37 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls lb # [ 237.013484] niks3-server[1154]: 2026/09/22 16:57:37 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo worker2 # [ 236.988449] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35652 worker2 # [ 237.000318] nix-daemon[843]: accepted connection from pid 1864, user nix-grpc-daemon (trusted) worker2 # [ 237.008432] nix-daemon[843]: reaped child process 1922, status = succeeded worker2 # [ 237.013834] nix-daemon[843]: accepted connection from pid 1864, user nix-grpc-daemon (trusted) client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051... client # farm-a> worker2-next: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-drain worker2 # [ 237.916340] nix-daemon[843]: reaped child process 1943, status = succeeded worker2 # [ 237.988159] nix-daemon[843]: accepted connection from pid 1959, user nix-grpc-daemon (trusted) worker2 # [ 238.006523] nix-daemon[843]: reaped child process 1962, status = succeeded lb # [ 238.043116] niks3-server[1154]: 2026/09/22 16:57:38 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 238.028465] nix-grpc-daemon[1924]: time=2026-09-22T16:57:38.674Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 238.030138] nix-grpc-daemon[1924]: time=2026-09-22T16:57:38.674Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)" lb # [ 238.090875] niks3-server[1154]: 2026/09/22 16:57:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign worker2 # [ 238.078294] nix-grpc-daemon[1924]: time=2026-09-22T16:57:38.724Z level=INFO msg="Uploading 1 narinfos" lb # [ 238.109344] niks3-server[1154]: 2026/09/22 16:57:38 INFO Signed narinfos id=81 count=1 lb # [ 238.121750] niks3-server[1154]: 2026/09/22 16:57:38 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv lb # [ 238.130836] niks3-server[1154]: 2026/09/22 16:57:38 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst lb # [ 238.143643] niks3-server[1154]: 2026/09/22 16:57:38 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls lb # [ 238.147878] niks3-server[1154]: 2026/09/22 16:57:38 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete lb # [ 238.152711] niks3-server[1154]: 2026/09/22 16:57:38 INFO Completed upload id=81 worker2 # [ 238.125640] nix-grpc-daemon[1924]: time=2026-09-22T16:57:38.772Z level=INFO msg="Upload complete. (299ms)" lb # [ 238.155287] niks3-server[1154]: 2026/09/22 16:57:38 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo worker2 # [ 238.141478] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35652 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=16 outputs=1 worker1 # [ 238.195983] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:38Z level=debug event=want client=7 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=0 worker1 # [ 238.196824] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:38Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 238.162842] nix-daemon[843]: reaped child process 1941, status = succeeded worker2 # [ 238.194179] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35652 worker2 # [ 238.209395] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:35652 worker2 # [ 238.225379] nix-grpc-daemon[1924]: time=2026-09-22T16:57:38.872Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'... worker2 # [ 238.236721] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35652 worker2 # [ 238.250884] nix-daemon[843]: accepted connection from pid 1864, user nix-grpc-daemon (trusted) worker2 # [ 238.408172] nix-daemon[843]: accepted connection from pid 1966, user nix-grpc-daemon (trusted) worker2 # [ 238.426718] nix-daemon[843]: reaped child process 1969, status = succeeded lb # [ 238.461985] niks3-server[1154]: 2026/09/22 16:57:39 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 238.444119] nix-grpc-daemon[1924]: time=2026-09-22T16:57:39.090Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 238.446184] nix-grpc-daemon[1924]: time=2026-09-22T16:57:39.090Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)" lb # [ 238.495039] niks3-server[1154]: 2026/09/22 16:57:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign worker2 # [ 238.476250] nix-grpc-daemon[1924]: time=2026-09-22T16:57:39.123Z level=INFO msg="Uploading 1 narinfos" lb # [ 238.510077] niks3-server[1154]: 2026/09/22 16:57:39 INFO Signed narinfos id=82 count=1 lb # [ 238.517864] niks3-server[1154]: 2026/09/22 16:57:39 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst lb # [ 238.530414] niks3-server[1154]: 2026/09/22 16:57:39 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls worker2 # [ 238.510133] nix-grpc-daemon[1924]: time=2026-09-22T16:57:39.157Z level=INFO msg="Upload complete. (242ms)" lb # [ 238.539438] niks3-server[1154]: 2026/09/22 16:57:39 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete lb # [ 238.545816] niks3-server[1154]: 2026/09/22 16:57:39 INFO Completed upload id=82 worker2 # [ 238.520324] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35652 duration_s=0 paths=1 nar_bytes_in=632 lb # [ 238.550017] niks3-server[1154]: 2026/09/22 16:57:39 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo worker2 # [ 238.536682] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35652 worker2 # [ 238.544556] nix-daemon[843]: accepted connection from pid 1864, user nix-grpc-daemon (trusted) worker2 # [ 238.550432] nix-daemon[843]: reaped child process 1964, status = succeeded client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051... worker2 # [ 238.564724] nix-daemon[843]: accepted connection from pid 1864, user nix-grpc-daemon (trusted) client # farm-b> worker2-next: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv lb # [ 239.152931] envoy[895]: conn peer=[2001:db8:1::1]:49684 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=897 tx=580 ms=30089 client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-drain worker2 # [ 239.604669] nix-daemon[843]: reaped child process 1973, status = succeeded worker2 # [ 239.720192] nix-daemon[843]: accepted connection from pid 1989, user nix-grpc-daemon (trusted) worker2 # [ 239.742656] nix-daemon[843]: reaped child process 1992, status = succeeded lb # [ 239.774055] niks3-server[1154]: 2026/09/22 16:57:40 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 239.760373] nix-grpc-daemon[1924]: time=2026-09-22T16:57:40.406Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 239.762249] nix-grpc-daemon[1924]: time=2026-09-22T16:57:40.406Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)" lb # [ 239.815962] niks3-server[1154]: 2026/09/22 16:57:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign worker2 # [ 239.797463] nix-grpc-daemon[1924]: time=2026-09-22T16:57:40.444Z level=INFO msg="Uploading 1 narinfos" lb # [ 239.833626] niks3-server[1154]: 2026/09/22 16:57:40 INFO Signed narinfos id=83 count=1 lb # [ 239.839821] niks3-server[1154]: 2026/09/22 16:57:40 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls lb # [ 239.847392] niks3-server[1154]: 2026/09/22 16:57:40 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker2 # [ 239.830954] nix-grpc-daemon[1924]: time=2026-09-22T16:57:40.478Z level=INFO msg="Upload complete. (252ms)" lb # [ 239.859838] niks3-server[1154]: 2026/09/22 16:57:40 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst lb # [ 239.866054] niks3-server[1154]: 2026/09/22 16:57:40 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete lb # [ 239.870469] niks3-server[1154]: 2026/09/22 16:57:40 INFO Completed upload id=83 worker2 # [ 239.841133] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35652 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=17 outputs=1 lb # [ 239.873556] niks3-server[1154]: 2026/09/22 16:57:40 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo worker1 # [ 239.906218] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:40Z level=debug event=want client=7 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=0 worker2 # [ 239.860587] nix-daemon[843]: reaped child process 1971, status = succeeded worker1 # [ 239.912683] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:40Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 239.864530] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35652 worker2 # [ 239.877994] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:35652 worker2 # [ 239.890678] nix-grpc-daemon[1924]: time=2026-09-22T16:57:40.537Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'... worker2 # [ 239.900822] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35652 worker2 # [ 239.913540] nix-daemon[843]: accepted connection from pid 1864, user nix-grpc-daemon (trusted) worker2 # [ 240.050975] nix-daemon[843]: accepted connection from pid 1996, user nix-grpc-daemon (trusted) worker2 # [ 240.074613] nix-daemon[843]: reaped child process 1999, status = succeeded lb # [ 240.105984] niks3-server[1154]: 2026/09/22 16:57:40 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 240.087248] nix-grpc-daemon[1924]: time=2026-09-22T16:57:40.734Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 240.088976] nix-grpc-daemon[1924]: time=2026-09-22T16:57:40.734Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)" lb # [ 240.136090] niks3-server[1154]: 2026/09/22 16:57:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign worker2 # [ 240.115227] nix-grpc-daemon[1924]: time=2026-09-22T16:57:40.762Z level=INFO msg="Uploading 1 narinfos" lb # [ 240.151178] niks3-server[1154]: 2026/09/22 16:57:40 INFO Signed narinfos id=84 count=1 lb # [ 240.158228] niks3-server[1154]: 2026/09/22 16:57:40 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls lb # [ 240.169407] niks3-server[1154]: 2026/09/22 16:57:40 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst worker2 # [ 240.147458] nix-grpc-daemon[1924]: time=2026-09-22T16:57:40.794Z level=INFO msg="Upload complete. (220ms)" lb # [ 240.175354] niks3-server[1154]: 2026/09/22 16:57:40 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete lb # [ 240.179243] niks3-server[1154]: 2026/09/22 16:57:40 INFO Completed upload id=84 lb # [ 240.184263] niks3-server[1154]: 2026/09/22 16:57:40 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo worker2 # [ 240.156302] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35652 duration_s=0 paths=1 nar_bytes_in=760 worker2 # [ 240.172538] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35652 worker2 # [ 240.178808] nix-daemon[843]: accepted connection from pid 1864, user nix-grpc-daemon (trusted) worker2 # [ 240.188351] nix-daemon[843]: reaped child process 1994, status = succeeded client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051... worker2 # [ 240.197425] nix-daemon[843]: accepted connection from pid 1864, user nix-grpc-daemon (trusted) client # farm-top> worker2-next: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-drain worker2 # [ 241.299122] nix-daemon[843]: reaped child process 2003, status = succeeded worker2 # [ 241.382384] nix-daemon[843]: accepted connection from pid 2020, user nix-grpc-daemon (trusted) worker2 # [ 241.406393] nix-daemon[843]: reaped child process 2023, status = succeeded lb # [ 241.433412] niks3-server[1154]: 2026/09/22 16:57:42 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 241.416856] nix-grpc-daemon[1924]: time=2026-09-22T16:57:42.063Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 241.420730] nix-grpc-daemon[1924]: time=2026-09-22T16:57:42.063Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)" lb # [ 241.479072] niks3-server[1154]: 2026/09/22 16:57:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign worker2 # [ 241.460288] nix-grpc-daemon[1924]: time=2026-09-22T16:57:42.107Z level=INFO msg="Uploading 1 narinfos" lb # [ 241.491621] niks3-server[1154]: 2026/09/22 16:57:42 INFO Signed narinfos id=85 count=1 lb # [ 241.500411] niks3-server[1154]: 2026/09/22 16:57:42 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls lb # [ 241.503738] niks3-server[1154]: 2026/09/22 16:57:42 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst lb # [ 241.514945] niks3-server[1154]: 2026/09/22 16:57:42 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker2 # [ 241.494391] nix-grpc-daemon[1924]: time=2026-09-22T16:57:42.141Z level=INFO msg="Upload complete. (285ms)" lb # [ 241.523512] niks3-server[1154]: 2026/09/22 16:57:42 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete lb # [ 241.526894] niks3-server[1154]: 2026/09/22 16:57:42 INFO Completed upload id=85 lb # [ 241.529439] niks3-server[1154]: 2026/09/22 16:57:42 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo worker2 # [ 241.509218] nix-grpc-daemon[1864]: ts=2026-09-22T16:57:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35652 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=18 outputs=1 worker2 # [ 241.527283] nix-daemon[843]: reaped child process 2001, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 5.34 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.19 seconds) worker1: must succeed: rm /nix/.rw-store/fill worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.16 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free' worker1 # [ 242.672717] nix-grpc-daemon[1595]: ts=2026-09-22T16:57:43Z level=info event=healthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.10 seconds) (finished: subtest: low disk drains a worker and builds go to the other, in 9.14 seconds) (finished: run the VM test script, in 243.99 seconds) test script finished in 244.60s cleanup kill QemuMachine (pid 49) client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) client # [2026-09-22T16:57:44Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-22T16:57:44Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-22T16:57:44Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 50) lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) lb # [2026-09-22T16:57:44Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-22T16:57:44Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-22T16:57:44Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 51) worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker1 # [2026-09-22T16:57:44Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-22T16:57:44Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-22T16:57:44Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 52) worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker2 # [2026-09-22T16:57:45Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-22T16:57:45Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-22T16:57:45Z INFO virtiofsd] Client disconnected, shutting down (finished: cleanup, in 1.48 seconds) additionally exposed symbols: client, lb, worker1, worker2, vlan1, start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh