tribuchet: building on eliza Machine state will be reset. To keep it, pass --keep-machine-state start all VLans (finished: start all VLans, in 0.00 seconds) Test will time out and terminate in 900.0 seconds run the VM test script start all VMs client: starting vm lb: starting vm worker1: starting vm worker2: starting vm worker2 # Disk image does not exist, creating the virtualisation disk image... worker2: QEMU running (pid 52) worker2 # Formatting '/build/vm-state-worker2/tmp.fK3UmTUHCK', fmt=raw size=1073741824 client # Disk image does not exist, creating the virtualisation disk image... worker1: QEMU running (pid 51) worker2 # mke2fs 1.47.4 (6-Mar-2025) worker2 # Discarding device blocks: 0/262144 done client # Formatting '/build/vm-state-client/tmp.Q15N65xSjo', fmt=raw size=1073741824 worker1 # Disk image does not exist, creating the virtualisation disk image... lb # Disk image does not exist, creating the virtualisation disk image... client # mke2fs 1.47.4 (6-Mar-2025) worker1 # Formatting '/build/vm-state-worker1/tmp.Pj2lpEIY6y', fmt=raw size=1073741824 lb: QEMU running (pid 50) client: QEMU running (pid 49) worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes lb # Formatting '/build/vm-state-lb/tmp.N9FHIroyDW', fmt=raw size=1073741824 worker1 # mke2fs 1.47.4 (6-Mar-2025) client # Discarding device blocks: 0/262144 done worker2 # Filesystem UUID: 203ff707-c759-4c3a-a3f9-d5fbb101f69b (finished: start all VMs, in 0.56 seconds) client # Creating filesystem with 262144 4k blocks and 65536 inodes worker2 # Superblock backups stored on blocks: client: waiting for unit network-addresses-eth1.service client # Filesystem UUID: 17878813-a2c4-493a-b0e7-fd3fa6804a97 worker2 # 32768, 98304, 163840, 229376 client: waiting for the VM to finish booting client # Superblock backups stored on blocks: client # 32768, 98304, 163840, 229376 worker1 # Discarding device blocks: 0/262144 done lb # mke2fs 1.47.4 (6-Mar-2025) worker2 # client # worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes lb # Discarding device blocks: 0/262144 done worker2 # Allocating group tables: 0/8 done client # Allocating group tables: 0/8 done worker1 # Filesystem UUID: 6fdfa4c4-3465-40d7-a9c6-192ad8c81d98 lb # Creating filesystem with 262144 4k blocks and 65536 inodes worker2 # Writing inode tables: 0/8 done client # Writing inode tables: 0/8 done worker1 # Superblock backups stored on blocks: lb # Filesystem UUID: 98fd74e6-f0d8-4597-b85e-8a4553334efc worker2 # Creating journal (8192 blocks): done client # Creating journal (8192 blocks): done worker1 # 32768, 98304, 163840, 229376 lb # Superblock backups stored on blocks: worker2 # Writing superblocks and filesystem accounting information: 0/8 done client # Writing superblocks and filesystem accounting information: 0/8 done worker1 # lb # 32768, 98304, 163840, 229376 worker2 # client # worker1 # Allocating group tables: 0/8 done lb # worker2 # Virtualisation disk image created. client # Virtualisation disk image created. worker1 # Writing inode tables: 0/8 done lb # Allocating group tables: 0/8 done worker2 # Starting virtiofs daemons... client # Starting virtiofs daemons... worker1 # Creating journal (8192 blocks): done lb # Writing inode tables: 0/8 done worker2 # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # Writing superblocks and filesystem accounting information: 0/8 done lb # Creating journal (8192 blocks): done worker2 # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # lb # Writing superblocks and filesystem accounting information: 0/8 done worker2 # [2026-09-23T05:12:11Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-23T05:12:11Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # Virtualisation disk image created. lb # worker2 # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # Starting virtiofs daemons... lb # Virtualisation disk image created. worker2 # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # Starting virtiofs daemons... worker2 # [2026-09-23T05:12:11Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-23T05:12:11Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-23T05:12:11Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-23T05:12:11Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-23T05:12:11Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-23T05:12:11Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-23T05:12:11Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-23T05:12:11Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T05:12:11Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-23T05:12:11Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-23T05:12:11Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-23T05:12:11Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-23T05:12:11Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-23T05:12:11Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-23T05:12:11Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-23T05:12:11Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T05:12:11Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-23T05:12:11Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T05:12:11Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-23T05:12:11Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T05:12:11Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-23T05:12:11Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-23T05:12:11Z INFO virtiofsd] Client connected, servicing requests worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker2 # [ 0.000000] KASLR enabled worker2 # [ 0.000000] random: crng init done worker2 # [ 0.000000] Machine model: linux,dummy-virt worker2 # [ 0.000000] efi: UEFI not found. worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker2 # [ 0.000000] Zone ranges: worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] DMA32 empty worker2 # [ 0.000000] Normal empty worker2 # [ 0.000000] Device empty worker2 # [ 0.000000] Movable zone start for each node worker2 # [ 0.000000] Early memory node ranges worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker2 # [ 0.000000] psci: probing for conduit method from DT. worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker2 # [ 0.000000] psci: Trusted OS migration not required worker2 # [ 0.000000] psci: SMC Calling Convention v1.1 worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker2 # [ 0.000000] Detected PIPT I-cache on CPU0 worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface worker2 # [ 0.000000] CPU features: detected: Spectre-v4 worker2 # [ 0.000000] CPU features: detected: Spectre-BHB worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker2 # [ 0.000000] alternatives: applying boot alternatives worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/zmhsivlz18fkqv0yqzlasss6r4v9smgp-nixos-system-worker2-test/init regInfo=/nix/store/3d1c8rkg6cnhhvdrq0ra4awlc1dyyzs4-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/3d1c8rkg6cnhhvdrq0ra4awlc1dyyzs4-closure-info/registration", will be passed to user space. worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker2 # [ 0.000000] software IO TLB: area num 1. worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker2 # [ 0.000000] Fallback order for Node 0: 0 worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker2 # [ 0.000000] Policy zone: DMA worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker2 # [ 0.000000] allocated 3145728 bytes of page_ext worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker2 # [ 0.000000] rcu: Hierarchical RCU implementation. worker2 # [ 0.000000] rcu: RCU event tracing is enabled. worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled. worker2 # [ 0.000000] Rude variant of Tasks RCU enabled. worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled. worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker2 # [ 0.000000] GICv3: 256 SPIs implemented client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented client # [ 0.000000] KASLR enabled client # [ 0.000000] random: crng init done worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq client # [ 0.000000] Machine model: linux,dummy-virt worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI client # [ 0.000000] efi: UEFI not found. worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) client # [ 0.000000] Zone ranges: client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) client # [ 0.000000] DMA32 empty client # [ 0.000000] Normal empty worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a20000 client # [ 0.000000] Device empty client # [ 0.000000] Movable zone start for each node worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a30000 client # [ 0.000000] Early memory node ranges worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). client # [ 0.000000] psci: probing for conduit method from DT. client # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs client # [ 0.000000] psci: Trusted OS migration not required worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns client # [ 0.000000] psci: SMC Calling Convention v1.1 worker2 # [ 0.000038] arm-pv: using stolen time PV client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker2 # [ 0.000817] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker2 # [ 0.001026] Console: colour dummy device 80x25 client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker2 # [ 0.001034] printk: legacy console [tty0] enabled client # [ 0.000000] Detected PIPT I-cache on CPU0 client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker2 # [ 0.001252] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) client # [ 0.000000] CPU features: detected: GICv3 CPU interface worker2 # [ 0.001259] pid_max: default: 32768 minimum: 301 client # [ 0.000000] CPU features: detected: Spectre-v4 worker2 # [ 0.001337] LSM: initializing lsm=capability,landlock,yama,bpf,ima client # [ 0.000000] CPU features: detected: Spectre-BHB worker2 # [ 0.001572] landlock: Up and running. worker2 # [ 0.001575] Yama: becoming mindful. client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker2 # [ 0.002318] LSM support for eBPF active client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker2 # [ 0.002465] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.000000] alternatives: applying boot alternatives worker2 # [ 0.002505] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.004538] rcu: Hierarchical SRCU implementation. worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker1 # [ 0.000000] KASLR enabled worker2 # [ 0.004545] rcu: Max phase no-delay instances is 1000. worker1 # [ 0.000000] random: crng init done worker2 # [ 0.005870] fsl-mc MSI: its@8080000 domain created worker1 # [ 0.000000] Machine model: linux,dummy-virt worker1 # [ 0.000000] efi: UEFI not found. worker2 # [ 0.005964] EFI services will not be available. worker2 # [ 0.006052] smp: Bringing up secondary CPUs ... client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/3zbq1wdsr2hpk2a2bj4l4l0k357dbi4h-nixos-system-client-test/init regInfo=/nix/store/9a83v96a2qvmk2ylwmy17pwh52xpkrhr-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker2 # [ 0.006062] smp: Brought up 1 node, 1 CPU worker2 # [ 0.006065] SMP: Total of 1 processors activated. worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.006072] CPU: All CPU(s) started at EL1 client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/9a83v96a2qvmk2ylwmy17pwh52xpkrhr-closure-info/registration", will be passed to user space. worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker1 # [ 0.000000] Zone ranges: worker2 # [ 0.006084] CPU features: detected: Branch Target Identification client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.006088] CPU features: detected: ARMv8.4 Translation Table Level worker1 # [ 0.000000] DMA32 empty client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.000000] Normal empty worker1 # [ 0.000000] Device empty worker2 # [ 0.006091] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker1 # [ 0.000000] Movable zone start for each node client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker1 # [ 0.000000] Early memory node ranges worker2 # [ 0.006095] CPU features: detected: Data cache clean to the PoU not required for I/D coherence client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] software IO TLB: area num 1. worker2 # [ 0.006099] CPU features: detected: Common not Private translations worker2 # [ 0.006102] CPU features: detected: CRC32 instructions worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker2 # [ 0.006105] CPU features: detected: Data cache clean to Point of Deep Persistence client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker1 # [ 0.000000] psci: probing for conduit method from DT. client # [ 0.000000] Fallback order for Node 0: 0 worker2 # [ 0.006108] CPU features: detected: Data cache clean to Point of Persistence worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware. client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker2 # [ 0.006111] CPU features: detected: Data independent timing control (DIT) client # [ 0.000000] Policy zone: DMA worker1 # [ 0.000000] psci: Trusted OS migration not required worker2 # [ 0.006115] CPU features: detected: E0PD client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker1 # [ 0.000000] psci: SMC Calling Convention v1.1 worker2 # [ 0.006117] CPU features: detected: Enhanced Counter Virtualization client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker2 # [ 0.006120] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) client # [ 0.000000] allocated 3145728 bytes of page_ext worker2 # [ 0.006124] CPU features: detected: Enhanced Virtualization Traps client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker1 # [ 0.000000] Detected PIPT I-cache on CPU0 worker2 # [ 0.006127] CPU features: detected: Fine Grained Traps client # [ 0.000000] ftrace: allocated 294 pages with 4 groups client # [ 0.000000] rcu: Hierarchical RCU implementation. worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker2 # [ 0.006131] CPU features: detected: Generic authentication (architected QARMA5 algorithm) client # [ 0.000000] rcu: RCU event tracing is enabled. worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface worker2 # [ 0.006136] CPU features: detected: RCpc load-acquire (LDAPR) worker1 # [ 0.000000] CPU features: detected: Spectre-v4 client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker2 # [ 0.006139] CPU features: detected: LSE atomic instructions worker1 # [ 0.000000] CPU features: detected: Spectre-BHB client # [ 0.000000] Trampoline variant of Tasks RCU enabled. worker2 # [ 0.006142] CPU features: detected: Privileged Access Never client # [ 0.000000] Rude variant of Tasks RCU enabled. worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker2 # [ 0.006144] CPU features: detected: PMUv3 client # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker2 # [ 0.006148] CPU features: detected: RAS Extension Support worker1 # [ 0.000000] alternatives: applying boot alternatives client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker2 # [ 0.006151] CPU features: detected: RASv1p1 Extension Support client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 worker2 # [ 0.006153] CPU features: detected: Random Number Generator worker2 # [ 0.006156] CPU features: detected: Speculation barrier (SB) client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.006159] CPU features: detected: Stage-2 Force Write-Back client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.006162] CPU features: detected: TLB range maintenance instructions worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/6c2avm42395n27rpri261y0pclhhiysh-nixos-system-worker1-test/init regInfo=/nix/store/3d6a9ard07ivq4f654pnmjwkx5sfygsp-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker2 # [ 0.006166] CPU features: detected: Speculative Store Bypassing Safe (SSBS) client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.006208] alternatives: applying system-wide alternatives client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 client # [ 0.000000] GICv3: 256 SPIs implemented worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/3d6a9ard07ivq4f654pnmjwkx5sfygsp-closure-info/registration", will be passed to user space. worker2 # [ 0.009282] CPU features: detected: BBM Level 2 without TLB conflict abort client # [ 0.000000] GICv3: 0 Extended SPIs implemented worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes client # [ 0.000000] Root IRQ handler: gic_handle_irq client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI worker2 # [ 0.009528] Memory: 1405616K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved) worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker2 # [ 0.009974] devtmpfs: initialized worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker2 # [ 0.011964] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker1 # [ 0.000000] software IO TLB: area num 1. worker2 # [ 0.011989] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker2 # [ 0.012192] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 lb # [ 0.000000] KASLR enabled worker2 # [ 0.012197] 0 pages in range for non-PLT usage lb # [ 0.000000] random: crng init done worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker2 # [ 0.012198] 508272 pages in range for PLT usage worker1 # [ 0.000000] Fallback order for Node 0: 0 worker2 # [ 0.012322] pinctrl core: initialized pinctrl subsystem lb # [ 0.000000] Machine model: linux,dummy-virt client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker2 # [ 0.014189] DMI not present or invalid. lb # [ 0.000000] efi: UEFI not found. client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker1 # [ 0.000000] Policy zone: DMA client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker2 # [ 0.018218] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. worker2 # [ 0.020659] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 lb # [ 0.000000] Zone ranges: client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker2 # [ 0.020854] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker1 # [ 0.000000] allocated 3145728 bytes of page_ext lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages lb # [ 0.000000] DMA32 empty worker2 # [ 0.021068] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations lb # [ 0.000000] Normal empty worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups lb # [ 0.000000] Device empty worker1 # [ 0.000000] rcu: Hierarchical RCU implementation. worker2 # [ 0.021120] audit: initializing netlink subsys (disabled) client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns lb # [ 0.000000] Movable zone start for each node worker1 # [ 0.000000] rcu: RCU event tracing is enabled. lb # [ 0.000000] Early memory node ranges worker2 # [ 0.021408] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000040] arm-pv: using stolen time PV worker2 # [ 0.021779] thermal_sys: Registered thermal governor 'fair_share' worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled. lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.021782] thermal_sys: Registered thermal governor 'bang_bang' worker1 # [ 0.000000] Rude variant of Tasks RCU enabled. client # [ 0.000635] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled. worker2 # [ 0.021785] thermal_sys: Registered thermal governor 'step_wise' client # [ 0.000844] Console: colour dummy device 80x25 lb # [ 0.000000] psci: probing for conduit method from DT. worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker2 # [ 0.021791] thermal_sys: Registered thermal governor 'user_space' client # [ 0.000853] printk: legacy console [tty0] enabled lb # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker2 # [ 0.021793] thermal_sys: Registered thermal governor 'power_allocator' lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 worker2 # [ 0.021821] cpuidle: using governor ladder client # [ 0.001054] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) lb # [ 0.000000] psci: Trusted OS migration not required worker2 # [ 0.021829] cpuidle: using governor menu worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.001061] pid_max: default: 32768 minimum: 301 lb # [ 0.000000] psci: SMC Calling Convention v1.1 worker2 # [ 0.022038] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. client # [ 0.001150] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) client # [ 0.001372] landlock: Up and running. worker2 # [ 0.022058] ASID allocator initialised with 65536 entries client # [ 0.001375] Yama: becoming mindful. worker2 # [ 0.023371] Serial: AMBA PL011 UART driver lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.002056] LSM support for eBPF active lb # [ 0.000000] Detected PIPT I-cache on CPU0 worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker2 # [ 0.029241] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 client # [ 0.002233] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker1 # [ 0.000000] GICv3: 256 SPIs implemented lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented client # [ 0.002267] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) lb # [ 0.000000] CPU features: detected: GICv3 CPU interface worker2 # [ 0.029408] printk: console [ttyAMA0] enabled worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq lb # [ 0.000000] CPU features: detected: Spectre-v4 client # [ 0.004301] rcu: Hierarchical SRCU implementation. worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI lb # [ 0.000000] CPU features: detected: Spectre-BHB client # [ 0.004307] rcu: Max phase no-delay instances is 1000. worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 client # [ 0.005613] fsl-mc MSI: its@8080000 domain created lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 client # [ 0.005710] EFI services will not be available. lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] client # [ 0.005798] smp: Bringing up secondary CPUs ... lb # [ 0.000000] alternatives: applying boot alternatives client # [ 0.005808] smp: Brought up 1 node, 1 CPU worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) client # [ 0.005811] SMP: Total of 1 processors activated. client # [ 0.005817] CPU: All CPU(s) started at EL1 worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) client # [ 0.005828] CPU features: detected: Branch Target Identification worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 client # [ 0.005832] CPU features: detected: ARMv8.4 Translation Table Level worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/kgjzg78npi7z07am2szqxkr41f06jp5w-nixos-system-lb-test/init regInfo=/nix/store/lcw0xq38k7svc2ifmp8wza1s56mbyq61-closure-info/registration console=ttyAMA0,115200n8 console=tty0 client # [ 0.005836] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. client # [ 0.005839] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker2 # [ 0.154082] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/lcw0xq38k7svc2ifmp8wza1s56mbyq61-closure-info/registration", will be passed to user space. client # [ 0.005843] CPU features: detected: Common not Private translations worker2 # [ 0.154107] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). client # [ 0.005847] CPU features: detected: CRC32 instructions lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker2 # [ 0.154113] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages client # [ 0.005850] CPU features: detected: Data cache clean to Point of Deep Persistence lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker2 # [ 0.154118] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page client # [ 0.005853] CPU features: detected: Data cache clean to Point of Persistence lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker2 # [ 0.154122] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages client # [ 0.005856] CPU features: detected: Data independent timing control (DIT) worker1 # [ 0.000037] arm-pv: using stolen time PV lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB client # [ 0.005859] CPU features: detected: E0PD worker2 # [ 0.154126] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.000000] software IO TLB: area num 1. client # [ 0.005862] CPU features: detected: Enhanced Counter Virtualization worker1 # [ 0.000486] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker2 # [ 0.154131] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker1 # [ 0.000698] Console: colour dummy device 80x25 worker2 # [ 0.154135] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page client # [ 0.005865] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) worker1 # [ 0.000706] printk: legacy console [tty0] enabled lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) client # [ 0.005869] CPU features: detected: Enhanced Virtualization Traps lb # [ 0.000000] Fallback order for Node 0: 0 worker1 # [ 0.000900] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) client # [ 0.005872] CPU features: detected: Fine Grained Traps lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 lb # [ 0.000000] Policy zone: DMA worker1 # [ 0.000907] pid_max: default: 32768 minimum: 301 client # [ 0.005876] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker1 # [ 0.000986] LSM: initializing lsm=capability,landlock,yama,bpf,ima lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker2 # [ 0.161933] fbcon: Taking over console client # [ 0.005881] CPU features: detected: RCpc load-acquire (LDAPR) worker1 # [ 0.001139] landlock: Up and running. worker2 # [ 0.161956] ACPI: Interpreter disabled. lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker1 # [ 0.001142] Yama: becoming mindful. client # [ 0.005884] CPU features: detected: LSE atomic instructions worker1 # [ 0.001694] LSM support for eBPF active lb # [ 0.000000] allocated 3145728 bytes of page_ext client # [ 0.005887] CPU features: detected: Privileged Access Never lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages worker1 # [ 0.001843] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.005889] CPU features: detected: PMUv3 lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups client # [ 0.005892] CPU features: detected: RAS Extension Support lb # [ 0.000000] rcu: Hierarchical RCU implementation. worker1 # [ 0.001884] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.005895] CPU features: detected: RASv1p1 Extension Support lb # [ 0.000000] rcu: RCU event tracing is enabled. worker1 # [ 0.003795] rcu: Hierarchical SRCU implementation. client # [ 0.005898] CPU features: detected: Random Number Generator worker1 # [ 0.003802] rcu: Max phase no-delay instances is 1000. lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker1 # [ 0.005107] fsl-mc MSI: its@8080000 domain created client # [ 0.005900] CPU features: detected: Speculation barrier (SB) lb # [ 0.000000] Trampoline variant of Tasks RCU enabled. worker1 # [ 0.005231] EFI services will not be available. client # [ 0.005903] CPU features: detected: Stage-2 Force Write-Back lb # [ 0.000000] Rude variant of Tasks RCU enabled. worker1 # [ 0.005321] smp: Bringing up secondary CPUs ... lb # [ 0.000000] Tracing variant of Tasks RCU enabled. client # [ 0.005906] CPU features: detected: TLB range maintenance instructions worker1 # [ 0.005330] smp: Brought up 1 node, 1 CPU worker1 # [ 0.005333] SMP: Total of 1 processors activated. lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. client # [ 0.005910] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker1 # [ 0.005339] CPU: All CPU(s) started at EL1 client # [ 0.005951] alternatives: applying system-wide alternatives lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 worker2 # [ 0.170526] iommu: Default domain type: Translated worker1 # [ 0.005350] CPU features: detected: Branch Target Identification worker2 # [ 0.170539] iommu: DMA domain TLB invalidation policy: strict mode client # [ 0.009163] CPU features: detected: BBM Level 2 without TLB conflict abort lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.005354] CPU features: detected: ARMv8.4 Translation Table Level worker2 # [ 0.172341] SCSI subsystem initialized worker1 # [ 0.005357] CPU features: detected: Instruction cache invalidation not required for I/D coherence lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.009383] Memory: 1405616K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved) client # [ 0.009802] devtmpfs: initialized worker2 # [ 0.173079] usbcore: registered new interface driver usbfs worker1 # [ 0.005361] CPU features: detected: Data cache clean to the PoU not required for I/D coherence lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.173144] usbcore: registered new interface driver hub client # [ 0.011720] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker1 # [ 0.005364] CPU features: detected: Common not Private translations lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker2 # [ 0.173161] usbcore: registered new device driver usb lb # [ 0.000000] GICv3: 256 SPIs implemented worker1 # [ 0.005368] CPU features: detected: CRC32 instructions client # [ 0.011750] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). lb # [ 0.000000] GICv3: 0 Extended SPIs implemented worker2 # [ 0.173446] pps_core: LinuxPPS API ver. 1 registered worker1 # [ 0.005370] CPU features: detected: Data cache clean to Point of Deep Persistence lb # [ 0.000000] Root IRQ handler: gic_handle_irq client # [ 0.011954] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI client # [ 0.011959] 0 pages in range for non-PLT usage worker2 # [ 0.173453] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker1 # [ 0.005374] CPU features: detected: Data cache clean to Point of Persistence worker2 # [ 0.173464] PTP clock support registered lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 client # [ 0.011960] 508272 pages in range for PLT usage worker2 # [ 0.173519] EDAC MC: Ver: 3.0.0 worker1 # [ 0.005377] CPU features: detected: Data independent timing control (DIT) client # [ 0.012081] pinctrl core: initialized pinctrl subsystem lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker1 # [ 0.005380] CPU features: detected: E0PD client # [ 0.012897] DMI not present or invalid. lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker2 # [ 0.178230] scmi_core: SCMI protocol bus registered worker1 # [ 0.005383] CPU features: detected: Enhanced Counter Virtualization client # [ 0.016617] NET: Registered PF_NETLINK/PF_ROUTE protocol family lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker2 # [ 0.179298] FPGA manager framework worker1 # [ 0.005386] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) client # [ 0.018961] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker2 # [ 0.180349] vgaarb: loaded worker1 # [ 0.005390] CPU features: detected: Enhanced Virtualization Traps client # [ 0.019166] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker1 # [ 0.005393] CPU features: detected: Fine Grained Traps worker2 # [ 0.181060] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 client # [ 0.019391] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker1 # [ 0.005396] CPU features: detected: Generic authentication (architected QARMA5 algorithm) lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 client # [ 0.019416] audit: initializing netlink subsys (disabled) worker1 # [ 0.005401] CPU features: detected: RCpc load-acquire (LDAPR) client # [ 0.020043] thermal_sys: Registered thermal governor 'fair_share' lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. worker1 # [ 0.005404] CPU features: detected: LSE atomic instructions client # [ 0.020046] thermal_sys: Registered thermal governor 'bang_bang' worker1 # [ 0.005407] CPU features: detected: Privileged Access Never lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker1 # [ 0.005410] CPU features: detected: PMUv3 client # [ 0.020050] thermal_sys: Registered thermal governor 'step_wise' lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker2 # [ 0.184807] VFS: Disk quotas dquot_6.6.0 client # [ 0.020053] thermal_sys: Registered thermal governor 'user_space' worker1 # [ 0.005413] CPU features: detected: RAS Extension Support worker1 # [ 0.005416] CPU features: detected: RASv1p1 Extension Support worker2 # [ 0.184858] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) client # [ 0.020058] thermal_sys: Registered thermal governor 'power_allocator' lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker1 # [ 0.005418] CPU features: detected: Random Number Generator client # [ 0.020090] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 worker1 # [ 0.005421] CPU features: detected: Speculation barrier (SB) lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns client # [ 0.020099] cpuidle: using governor ladder lb # [ 0.000030] arm-pv: using stolen time PV worker1 # [ 0.005424] CPU features: detected: Stage-2 Force Write-Back client # [ 0.020105] cpuidle: using governor menu worker1 # [ 0.005426] CPU features: detected: TLB range maintenance instructions client # [ 0.020336] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. lb # [ 0.000467] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker2 # [ 0.189054] netfs: FS-Cache loaded client # [ 0.020355] ASID allocator initialised with 65536 entries worker1 # [ 0.005432] CPU features: detected: Speculative Store Bypassing Safe (SSBS) lb # [ 0.000663] Console: colour dummy device 80x25 client # [ 0.021624] Serial: AMBA PL011 UART driver worker2 # [ 0.189716] pnp: PnP ACPI: disabled worker1 # [ 0.005473] alternatives: applying system-wide alternatives lb # [ 0.000674] printk: legacy console [tty0] enabled worker1 # [ 0.008469] CPU features: detected: BBM Level 2 without TLB conflict abort client # [ 0.027126] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 lb # [ 0.000867] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) client # [ 0.027313] printk: console [ttyAMA0] enabled lb # [ 0.000874] pid_max: default: 32768 minimum: 301 worker1 # [ 0.008674] Memory: 1405616K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved) lb # [ 0.000950] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker1 # [ 0.009093] devtmpfs: initialized lb # [ 0.001134] landlock: Up and running. lb # [ 0.001137] Yama: becoming mindful. worker1 # [ 0.011003] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) lb # [ 0.001657] LSM support for eBPF active worker1 # [ 0.011027] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). lb # [ 0.001808] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker1 # [ 0.011208] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL lb # [ 0.001842] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker1 # [ 0.011212] 0 pages in range for non-PLT usage lb # [ 0.003780] rcu: Hierarchical SRCU implementation. worker1 # [ 0.011214] 508272 pages in range for PLT usage lb # [ 0.003786] rcu: Max phase no-delay instances is 1000. worker2 # [ 0.193678] NET: Registered PF_INET protocol family worker1 # [ 0.011331] pinctrl core: initialized pinctrl subsystem lb # [ 0.005089] fsl-mc MSI: its@8080000 domain created worker1 # [ 0.012130] DMI not present or invalid. worker2 # [ 0.193981] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) lb # [ 0.005190] EFI services will not be available. worker1 # [ 0.016042] NET: Registered PF_NETLINK/PF_ROUTE protocol family lb # [ 0.005274] smp: Bringing up secondary CPUs ... lb # [ 0.005283] smp: Brought up 1 node, 1 CPU worker1 # [ 0.018363] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations lb # [ 0.005286] SMP: Total of 1 processors activated. worker1 # [ 0.018552] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations client # [ 0.150905] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages lb # [ 0.005291] CPU: All CPU(s) started at EL1 lb # [ 0.005301] CPU features: detected: Branch Target Identification client # [ 0.150927] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker1 # [ 0.018764] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations lb # [ 0.005306] CPU features: detected: ARMv8.4 Translation Table Level worker1 # [ 0.018789] audit: initializing netlink subsys (disabled) client # [ 0.150932] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages client # [ 0.150937] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page lb # [ 0.005309] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker1 # [ 0.019378] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.150941] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages lb # [ 0.005313] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker1 # [ 0.019381] thermal_sys: Registered thermal governor 'bang_bang' client # [ 0.150946] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.005316] CPU features: detected: Common not Private translations worker1 # [ 0.019385] thermal_sys: Registered thermal governor 'step_wise' client # [ 0.150950] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages lb # [ 0.005319] CPU features: detected: CRC32 instructions worker1 # [ 0.019388] thermal_sys: Registered thermal governor 'user_space' client # [ 0.150954] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page lb # [ 0.005322] CPU features: detected: Data cache clean to Point of Deep Persistence worker1 # [ 0.019393] thermal_sys: Registered thermal governor 'power_allocator' lb # [ 0.005326] CPU features: detected: Data cache clean to Point of Persistence worker1 # [ 0.019429] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 worker1 # [ 0.019438] cpuidle: using governor ladder lb # [ 0.005329] CPU features: detected: Data independent timing control (DIT) client # [ 0.158651] fbcon: Taking over console worker1 # [ 0.019444] cpuidle: using governor menu lb # [ 0.005332] CPU features: detected: E0PD client # [ 0.158668] ACPI: Interpreter disabled. worker1 # [ 0.019631] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. lb # [ 0.005334] CPU features: detected: Enhanced Counter Virtualization worker1 # [ 0.019648] ASID allocator initialised with 65536 entries lb # [ 0.005337] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) worker1 # [ 0.020893] Serial: AMBA PL011 UART driver lb # [ 0.005341] CPU features: detected: Enhanced Virtualization Traps worker1 # [ 0.026417] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 lb # [ 0.005344] CPU features: detected: Fine Grained Traps worker1 # [ 0.026588] printk: console [ttyAMA0] enabled lb # [ 0.005348] CPU features: detected: Generic authentication (architected QARMA5 algorithm) lb # [ 0.005353] CPU features: detected: RCpc load-acquire (LDAPR) lb # [ 0.005356] CPU features: detected: LSE atomic instructions lb # [ 0.005359] CPU features: detected: Privileged Access Never lb # [ 0.005362] CPU features: detected: PMUv3 lb # [ 0.005365] CPU features: detected: RAS Extension Support lb # [ 0.005367] CPU features: detected: RASv1p1 Extension Support client # [ 0.167285] iommu: Default domain type: Translated lb # [ 0.005370] CPU features: detected: Random Number Generator client # [ 0.167298] iommu: DMA domain TLB invalidation policy: strict mode lb # [ 0.005372] CPU features: detected: Speculation barrier (SB) client # [ 0.167706] SCSI subsystem initialized lb # [ 0.005375] CPU features: detected: Stage-2 Force Write-Back client # [ 0.169859] usbcore: registered new interface driver usbfs lb # [ 0.005378] CPU features: detected: TLB range maintenance instructions client # [ 0.169902] usbcore: registered new interface driver hub lb # [ 0.005384] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker1 # [ 0.148376] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages client # [ 0.169918] usbcore: registered new device driver usb lb # [ 0.005423] alternatives: applying system-wide alternatives client # [ 0.170241] pps_core: LinuxPPS API ver. 1 registered worker1 # [ 0.148396] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page lb # [ 0.008424] CPU features: detected: BBM Level 2 without TLB conflict abort worker1 # [ 0.148402] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages client # [ 0.170252] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti client # [ 0.170262] PTP clock support registered worker1 # [ 0.148406] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page client # [ 0.170327] EDAC MC: Ver: 3.0.0 lb # [ 0.008589] Memory: 1405632K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved) worker1 # [ 0.148411] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages lb # [ 0.009019] devtmpfs: initialized client # [ 0.175187] scmi_core: SCMI protocol bus registered worker1 # [ 0.148415] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.010920] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) client # [ 0.176237] FPGA manager framework worker1 # [ 0.148419] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages client # [ 0.177292] vgaarb: loaded lb # [ 0.010944] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker1 # [ 0.148424] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page lb # [ 0.011154] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL client # [ 0.177968] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.011158] 0 pages in range for non-PLT usage lb # [ 0.011160] 508272 pages in range for PLT usage worker1 # [ 0.156107] fbcon: Taking over console lb # [ 0.011291] pinctrl core: initialized pinctrl subsystem worker1 # [ 0.156127] ACPI: Interpreter disabled. lb # [ 0.012116] DMI not present or invalid. lb # [ 0.015589] NET: Registered PF_NETLINK/PF_ROUTE protocol family client # [ 0.181455] VFS: Disk quotas dquot_6.6.0 lb # [ 0.018085] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations client # [ 0.181495] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) lb # [ 0.018288] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations lb # [ 0.018495] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker2 # [ 0.226661] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) lb # [ 0.018523] audit: initializing netlink subsys (disabled) client # [ 0.185618] netfs: FS-Cache loaded worker2 # [ 0.226719] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) lb # [ 0.019131] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.185762] pnp: PnP ACPI: disabled lb # [ 0.019133] thermal_sys: Registered thermal governor 'bang_bang' worker2 # [ 0.226774] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) lb # [ 0.019137] thermal_sys: Registered thermal governor 'step_wise' worker2 # [ 0.226866] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) lb # [ 0.019139] thermal_sys: Registered thermal governor 'user_space' worker1 # [ 0.164701] iommu: Default domain type: Translated worker2 # [ 0.227016] TCP: Hash tables configured (established 16384 bind 16384) worker1 # [ 0.164713] iommu: DMA domain TLB invalidation policy: strict mode lb # [ 0.019144] thermal_sys: Registered thermal governor 'power_allocator' worker1 # [ 0.165164] SCSI subsystem initialized worker2 # [ 0.227122] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) lb # [ 0.019170] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 worker2 # [ 0.227168] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.019179] cpuidle: using governor ladder worker1 # [ 0.167171] usbcore: registered new interface driver usbfs lb # [ 0.019184] cpuidle: using governor menu worker2 # [ 0.227208] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.167204] usbcore: registered new interface driver hub worker2 # [ 0.227326] NET: Registered PF_UNIX/PF_LOCAL protocol family lb # [ 0.019371] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker1 # [ 0.167220] usbcore: registered new device driver usb worker2 # [ 0.227350] NET: Registered PF_XDP protocol family lb # [ 0.019387] ASID allocator initialised with 65536 entries worker1 # [ 0.167510] pps_core: LinuxPPS API ver. 1 registered worker2 # [ 0.227371] PCI: CLS 0 bytes, default 64 lb # [ 0.020635] Serial: AMBA PL011 UART driver worker2 # [ 0.227628] Trying to unpack rootfs image as initramfs... worker1 # [ 0.167517] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti client # [ 0.190061] NET: Registered PF_INET protocol family worker1 # [ 0.167527] PTP clock support registered lb # [ 0.026087] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 worker1 # [ 0.167578] EDAC MC: Ver: 3.0.0 lb # [ 0.026257] printk: console [ttyAMA0] enabled client # [ 0.190352] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker1 # [ 0.172238] scmi_core: SCMI protocol bus registered worker1 # [ 0.173248] FPGA manager framework worker1 # [ 0.174231] vgaarb: loaded worker1 # [ 0.174875] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.149140] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker1 # [ 0.179794] VFS: Disk quotas dquot_6.6.0 lb # [ 0.149168] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker1 # [ 0.179846] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) worker2 # [ 0.249148] kvm [1]: HYP mode not available lb # [ 0.149173] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages worker1 # [ 0.181632] netfs: FS-Cache loaded worker1 # [ 0.181759] pnp: PnP ACPI: disabled lb # [ 0.149178] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page lb # [ 0.149182] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages lb # [ 0.149186] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.149191] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages lb # [ 0.149195] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker1 # [ 0.185759] NET: Registered PF_INET protocol family lb # [ 0.157161] fbcon: Taking over console worker1 # [ 0.186038] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) lb # [ 0.157182] ACPI: Interpreter disabled. lb # [ 0.166263] iommu: Default domain type: Translated lb # [ 0.166275] iommu: DMA domain TLB invalidation policy: strict mode lb # [ 0.166669] SCSI subsystem initialized lb # [ 0.168754] usbcore: registered new interface driver usbfs client # [ 0.221761] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) lb # [ 0.168788] usbcore: registered new interface driver hub client # [ 0.221821] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) lb # [ 0.168804] usbcore: registered new device driver usb lb # [ 0.169102] pps_core: LinuxPPS API ver. 1 registered client # [ 0.221855] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) client # [ 0.221935] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) lb # [ 0.169109] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti lb # [ 0.169120] PTP clock support registered lb # [ 0.169171] EDAC MC: Ver: 3.0.0 lb # [ 0.173997] scmi_core: SCMI protocol bus registered lb # [ 0.175044] FPGA manager framework lb # [ 0.176075] vgaarb: loaded lb # [ 0.176718] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.181100] VFS: Disk quotas dquot_6.6.0 client # [ 0.226241] TCP: Hash tables configured (established 16384 bind 16384) lb # [ 0.181140] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) client # [ 0.226345] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) client # [ 0.226387] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.226425] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.226546] NET: Registered PF_UNIX/PF_LOCAL protocol family client # [ 0.226588] NET: Registered PF_XDP protocol family lb # [ 0.185127] netfs: FS-Cache loaded lb # [ 0.185282] pnp: PnP ACPI: disabled client # [ 0.226610] PCI: CLS 0 bytes, default 64 client # [ 0.226906] Trying to unpack rootfs image as initramfs... lb # [ 0.189390] NET: Registered PF_INET protocol family worker1 # [ 0.216090] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) lb # [ 0.189673] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker1 # [ 0.216141] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker1 # [ 0.216172] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) worker1 # [ 0.216249] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker1 # [ 0.216395] TCP: Hash tables configured (established 16384 bind 16384) worker1 # [ 0.216499] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker1 # [ 0.216535] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.216572] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.216677] NET: Registered PF_UNIX/PF_LOCAL protocol family client # [ 0.246019] kvm [1]: HYP mode not available worker1 # [ 0.216700] NET: Registered PF_XDP protocol family worker1 # [ 0.216719] PCI: CLS 0 bytes, default 64 worker1 # [ 0.216972] Trying to unpack rootfs image as initramfs... worker1 # [ 0.234537] kvm [1]: HYP mode not available lb # [ 0.220098] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) lb # [ 0.220153] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) lb # [ 0.220188] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) lb # [ 0.220267] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) lb # [ 0.220411] TCP: Hash tables configured (established 16384 bind 16384) lb # [ 0.220507] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) lb # [ 0.220546] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.220582] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.220692] NET: Registered PF_UNIX/PF_LOCAL protocol family lb # [ 0.229115] NET: Registered PF_XDP protocol family lb # [ 0.229143] PCI: CLS 0 bytes, default 64 lb # [ 0.229412] Trying to unpack rootfs image as initramfs... lb # [ 0.238577] kvm [1]: HYP mode not available worker2 # [ 0.389774] Initialise system trusted keyrings worker2 # [ 0.390568] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker2 # [ 0.391998] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker2 # [ 0.392816] 9p: Installing v9fs 9p2000 file system support worker1 # [ 0.348629] Initialise system trusted keyrings worker1 # [ 0.349493] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker2 # [ 0.421773] Key type asymmetric registered worker2 # [ 0.421802] Asymmetric key parser 'x509' registered worker1 # [ 0.350863] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker2 # [ 0.421890] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker2 # [ 0.424150] io scheduler mq-deadline registered worker2 # [ 0.424164] io scheduler kyber registered worker1 # [ 0.351699] 9p: Installing v9fs 9p2000 file system support lb # [ 0.339383] Initialise system trusted keyrings worker2 # [ 0.437918] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker2 # [ 0.438577] ledtrig-cpu: registered to indicate activity on CPUs worker2 # [ 0.438991] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: lb # [ 0.344831] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker2 # [ 0.439013] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 lb # [ 0.346345] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker2 # [ 0.439027] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker2 # [ 0.439035] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 lb # [ 0.347210] 9p: Installing v9fs 9p2000 file system support worker2 # [ 0.439059] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker1 # [ 0.379834] Key type asymmetric registered worker2 # [ 0.439086] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker1 # [ 0.379865] Asymmetric key parser 'x509' registered worker2 # [ 0.439168] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker1 # [ 0.379954] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker2 # [ 0.439179] pci_bus 0000:00: root bus resource [bus 00-ff] worker1 # [ 0.382291] io scheduler mq-deadline registered worker1 # [ 0.382305] io scheduler kyber registered worker2 # [ 0.439185] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker2 # [ 0.439191] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker2 # [ 0.439196] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker2 # [ 0.439268] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker2 # [ 0.439742] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.439935] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.439955] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.439989] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.440009] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.440468] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker2 # [ 0.440651] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.440668] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.440699] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.441258] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker2 # [ 0.441461] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] worker2 # [ 0.441480] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.441515] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.442003] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.442209] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.368003] Key type asymmetric registered worker2 # [ 0.442227] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.368035] Asymmetric key parser 'x509' registered worker2 # [ 0.442258] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.368112] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker2 # [ 0.442280] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.423680] Initialise system trusted keyrings worker2 # [ 0.442769] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint client # [ 0.424548] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker2 # [ 0.442965] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.395993] pl061_gpio 9030000.pl061: PL061 GPIO chip registered client # [ 0.425945] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker2 # [ 0.442998] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.396615] ledtrig-cpu: registered to indicate activity on CPUs worker2 # [ 0.443464] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker1 # [ 0.397028] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker2 # [ 0.443649] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.443680] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.397053] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker2 # [ 0.444076] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker1 # [ 0.397070] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 lb # [ 0.376799] io scheduler mq-deadline registered client # [ 0.426839] 9p: Installing v9fs 9p2000 file system support lb # [ 0.376824] io scheduler kyber registered worker2 # [ 0.444273] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker1 # [ 0.397080] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker2 # [ 0.444571] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.397104] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker2 # [ 0.444761] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.444792] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.397129] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker1 # [ 0.397245] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker1 # [ 0.397256] pci_bus 0000:00: root bus resource [bus 00-ff] worker1 # [ 0.397263] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker1 # [ 0.397268] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.397273] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.397339] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.397804] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.397997] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.398014] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.398043] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.398060] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.388830] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker1 # [ 0.398533] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.398725] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.398741] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.398770] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.399280] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker1 # [ 0.399473] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] worker1 # [ 0.399491] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.489464] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.399522] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.489694] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.390356] ledtrig-cpu: registered to indicate activity on CPUs worker2 # [ 0.489733] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.400001] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.390805] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker1 # [ 0.400191] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.490231] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.400207] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.490429] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.390827] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker1 # [ 0.400237] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.490461] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.390840] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker1 # [ 0.400256] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.490947] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint lb # [ 0.390849] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker1 # [ 0.400720] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker2 # [ 0.491262] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker1 # [ 0.400915] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.491286] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.390872] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits client # [ 0.455042] Key type asymmetric registered worker1 # [ 0.400945] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.491319] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.455080] Asymmetric key parser 'x509' registered lb # [ 0.390903] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] client # [ 0.455174] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker1 # [ 0.401438] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker2 # [ 0.491815] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.391000] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 client # [ 0.457374] io scheduler mq-deadline registered worker2 # [ 0.492003] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.401634] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.391014] pci_bus 0000:00: root bus resource [bus 00-ff] client # [ 0.457387] io scheduler kyber registered worker2 # [ 0.492021] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.401665] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.391020] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker2 # [ 0.492053] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.391025] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.402118] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker2 # [ 0.492683] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker1 # [ 0.402311] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] lb # [ 0.391030] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker2 # [ 0.492697] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker1 # [ 0.402568] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.391110] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.402758] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.492703] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned lb # [ 0.391562] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.402788] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.492752] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned lb # [ 0.391750] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.391766] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.492805] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned lb # [ 0.391796] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.492855] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned lb # [ 0.391812] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.492910] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned lb # [ 0.392282] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker2 # [ 0.492959] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned lb # [ 0.392468] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.493012] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned lb # [ 0.392484] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.493117] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned lb # [ 0.392517] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.493164] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned lb # [ 0.393012] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint lb # [ 0.393200] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] worker2 # [ 0.493218] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned lb # [ 0.393217] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.493304] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned lb # [ 0.393249] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.493356] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned lb # [ 0.393714] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.493379] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned lb # [ 0.393912] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.493403] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned lb # [ 0.393929] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.493429] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned lb # [ 0.393960] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.493452] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned lb # [ 0.393979] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.493475] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned lb # [ 0.394457] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker2 # [ 0.493499] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned client # [ 0.471175] pl061_gpio 9030000.pl061: PL061 GPIO chip registered lb # [ 0.394647] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.447460] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.493528] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned client # [ 0.471807] ledtrig-cpu: registered to indicate activity on CPUs worker1 # [ 0.447686] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.394679] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.493553] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned client # [ 0.472257] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker1 # [ 0.447720] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.395199] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker2 # [ 0.493578] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned client # [ 0.472280] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 lb # [ 0.395393] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.448196] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.493602] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned lb # [ 0.395423] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.472295] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker1 # [ 0.448388] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.493626] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned worker1 # [ 0.448418] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.493648] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned lb # [ 0.395824] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint client # [ 0.472305] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker2 # [ 0.493671] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned lb # [ 0.396012] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker1 # [ 0.448893] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint client # [ 0.472336] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker2 # [ 0.493694] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned lb # [ 0.396265] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.449245] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] client # [ 0.472359] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker1 # [ 0.449268] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.493716] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned lb # [ 0.396456] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.493739] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned worker1 # [ 0.449301] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.472467] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 lb # [ 0.396486] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.493762] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned client # [ 0.472478] pci_bus 0000:00: root bus resource [bus 00-ff] worker1 # [ 0.449803] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker2 # [ 0.493792] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] client # [ 0.472485] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker1 # [ 0.449999] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.493803] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] client # [ 0.472490] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.450016] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.493807] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] client # [ 0.472495] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.450046] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.494679] pci 0000:00:07.0: enabling device (0000 -> 0002) worker1 # [ 0.450642] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned client # [ 0.472570] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.450656] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned client # [ 0.473062] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.473254] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.450662] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned client # [ 0.473273] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.450710] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned client # [ 0.473304] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.450759] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned lb # [ 0.437122] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.473320] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.450811] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned lb # [ 0.437343] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.473810] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.437373] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.474039] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] client # [ 0.474057] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.437847] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.474087] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.438037] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.450861] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned lb # [ 0.438068] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.474554] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker1 # [ 0.450938] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned client # [ 0.474743] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] lb # [ 0.438546] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker1 # [ 0.450991] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned client # [ 0.474760] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.438853] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] lb # [ 0.438871] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.474790] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.451039] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned lb # [ 0.438901] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.475287] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.451086] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned client # [ 0.475480] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.439422] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.475496] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.451136] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned lb # [ 0.439617] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.439634] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.475526] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.451211] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned lb # [ 0.439664] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.475546] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.451260] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned lb # [ 0.440259] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned client # [ 0.476022] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker1 # [ 0.451281] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned lb # [ 0.440270] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned client # [ 0.476243] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.451304] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned lb # [ 0.440275] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned client # [ 0.476276] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.451325] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned lb # [ 0.440323] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned client # [ 0.476800] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker1 # [ 0.451347] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned client # [ 0.476996] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.440371] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker1 # [ 0.451369] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned client # [ 0.477027] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.440418] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker1 # [ 0.451390] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned client # [ 0.477463] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint lb # [ 0.440469] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker1 # [ 0.451418] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned client # [ 0.477652] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] lb # [ 0.440517] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker1 # [ 0.451440] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned client # [ 0.477926] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.451462] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned lb # [ 0.440565] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker1 # [ 0.451484] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned lb # [ 0.440613] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker1 # [ 0.451506] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned lb # [ 0.440661] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker1 # [ 0.451528] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned lb # [ 0.440707] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned worker2 # [ 0.564418] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker1 # [ 0.451549] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned worker1 # [ 0.451571] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned worker1 # [ 0.451592] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned worker1 # [ 0.451613] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned worker1 # [ 0.451637] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned worker1 # [ 0.451664] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] worker1 # [ 0.451673] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] client # [ 0.518219] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.451678] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] client # [ 0.518276] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.452533] pci 0000:00:07.0: enabling device (0000 -> 0002) client # [ 0.518831] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.519031] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.519066] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.468941] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned client # [ 0.519549] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.469014] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned client # [ 0.519750] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.469038] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned client # [ 0.519782] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.469061] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker2 # [ 0.574728] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) lb # [ 0.469084] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned client # [ 0.520284] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint client # [ 0.520583] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] lb # [ 0.469107] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned client # [ 0.520603] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.469132] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned client # [ 0.520633] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.469155] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned lb # [ 0.469183] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned client # [ 0.521133] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.521328] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.469205] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned client # [ 0.521345] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.469229] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned client # [ 0.521375] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.469253] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker2 # [ 0.579271] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) lb # [ 0.469276] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned lb # [ 0.469298] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned worker2 # [ 0.581745] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) lb # [ 0.469318] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned worker2 # [ 0.584137] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) lb # [ 0.469341] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned lb # [ 0.469363] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned lb # [ 0.469384] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned lb # [ 0.469406] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned lb # [ 0.469438] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.469448] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] lb # [ 0.469453] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] lb # [ 0.470299] pci 0000:00:07.0: enabling device (0000 -> 0002) client # [ 0.542097] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned client # [ 0.542127] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned client # [ 0.542132] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned client # [ 0.542229] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker2 # [ 0.594788] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) client # [ 0.542286] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker2 # [ 0.596910] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker1 # [ 0.526276] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) client # [ 0.542335] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned client # [ 0.542387] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned client # [ 0.542436] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned client # [ 0.542489] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned client # [ 0.542537] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned client # [ 0.542585] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned client # [ 0.542636] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned client # [ 0.542733] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned client # [ 0.542822] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned client # [ 0.542846] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker2 # [ 0.607260] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) client # [ 0.542868] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned client # [ 0.542891] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned client # [ 0.542918] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker1 # [ 0.536604] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) client # [ 0.542940] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned client # [ 0.542963] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned client # [ 0.542992] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned client # [ 0.543016] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned client # [ 0.543039] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned client # [ 0.543062] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned client # [ 0.543087] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned client # [ 0.543110] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned lb # [ 0.515963] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) client # [ 0.543136] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned client # [ 0.543158] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned client # [ 0.543180] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned client # [ 0.543203] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned client # [ 0.543226] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned lb # [ 0.519472] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker1 # [ 0.547624] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.549946] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) client # [ 0.543255] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] client # [ 0.543266] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] client # [ 0.543271] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] client # [ 0.544136] pci 0000:00:07.0: enabling device (0000 -> 0002) worker2 # [ 0.617440] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) worker2 # [ 0.619825] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker1 # [ 0.554219] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) worker2 # [ 0.623448] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) lb # [ 0.525841] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.562389] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) lb # [ 0.537408] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) lb # [ 0.539640] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) worker2 # [ 0.644869] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled client # [ 0.596276] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker1 # [ 0.572434] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker2 # [ 0.647824] msm_serial: driver initialized worker2 # [ 0.647972] SuperH (H)SCI(F) driver initialized worker2 # [ 0.648029] STM32 USART driver initialized worker1 # [ 0.576695] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker1 # [ 0.578713] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) lb # [ 0.549963] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) lb # [ 0.552529] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) client # [ 0.606735] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) lb # [ 0.563680] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker1 # [ 0.592437] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) lb # [ 0.566586] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) worker1 # [ 0.595895] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) client # [ 0.618312] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) client # [ 0.620485] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) client # [ 0.622685] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) client # [ 0.624995] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) worker1 # [ 0.609137] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker2 # [ 0.683205] loop: module loaded lb # [ 0.576891] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker2 # [ 0.683429] virtio_blk virtio2: 1/0/0 default/read/poll queues lb # [ 0.579439] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker1 # [ 0.612002] msm_serial: driver initialized worker1 # [ 0.612149] SuperH (H)SCI(F) driver initialized worker2 # [ 0.684498] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) client # [ 0.635328] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker1 # [ 0.612204] STM32 USART driver initialized client # [ 0.637439] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) lb # [ 0.590061] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled lb # [ 0.592872] msm_serial: driver initialized client # [ 0.648832] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) lb # [ 0.593110] SuperH (H)SCI(F) driver initialized lb # [ 0.593172] STM32 USART driver initialized worker2 # [ 0.697987] megasas: 07.734.00.00-rc1 worker2 # [ 0.698897] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] client # [ 0.655143] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) client # [ 0.657730] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker2 # [ 0.710493] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.710554] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.712494] Using buffer write method worker2 # [ 0.712621] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker1 # [ 0.647881] loop: module loaded worker1 # [ 0.648108] virtio_blk virtio2: 1/0/0 default/read/poll queues worker1 # [ 0.648941] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) client # [ 0.675482] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker2 # [ 0.726516] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.726574] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.728476] Using buffer write method worker1 # [ 0.655649] megasas: 07.734.00.00-rc1 worker2 # [ 0.728522] Concatenating MTD devices: worker2 # [ 0.728526] (0): "0.flash" worker1 # [ 0.656444] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] worker2 # [ 0.728530] (1): "0.flash" worker2 # [ 0.728543] into device "0.flash" lb # [ 0.630085] loop: module loaded lb # [ 0.630286] virtio_blk virtio2: 1/0/0 default/read/poll queues client # [ 0.683520] msm_serial: driver initialized lb # [ 0.631185] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) client # [ 0.683715] SuperH (H)SCI(F) driver initialized client # [ 0.683784] STM32 USART driver initialized worker1 # [ 0.667713] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.667755] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.670316] Using buffer write method worker1 # [ 0.670390] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] lb # [ 0.641516] megasas: 07.734.00.00-rc1 lb # [ 0.642313] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] lb # [ 0.644691] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.675624] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.675653] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.647577] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.677303] Using buffer write method worker1 # [ 0.677332] Concatenating MTD devices: worker1 # [ 0.677336] (0): "0.flash" worker1 # [ 0.677340] (1): "0.flash" worker1 # [ 0.677351] into device "0.flash" lb # [ 0.653396] Using buffer write method lb # [ 0.653496] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] lb # [ 0.656676] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.656705] Intel/Sharp Extended Query Table at 0x0031 client # [ 0.721420] loop: module loaded client # [ 0.721666] virtio_blk virtio2: 1/0/0 default/read/poll queues lb # [ 0.670460] Using buffer write method lb # [ 0.670526] Concatenating MTD devices: lb # [ 0.670531] (0): "0.flash" lb # [ 0.670535] (1): "0.flash" lb # [ 0.670541] into device "0.flash" client # [ 0.730882] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) client # [ 0.735892] megasas: 07.734.00.00-rc1 client # [ 0.736925] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] client # [ 0.740898] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 client # [ 0.740935] Intel/Sharp Extended Query Table at 0x0031 client # [ 0.751564] Using buffer write method client # [ 0.751678] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] client # [ 0.753700] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 client # [ 0.753746] Intel/Sharp Extended Query Table at 0x0031 client # [ 0.767170] Using buffer write method client # [ 0.767224] Concatenating MTD devices: client # [ 0.767228] (0): "0.flash" client # [ 0.767232] (1): "0.flash" client # [ 0.767244] into device "0.flash" worker2 # [ 1.030072] Freeing initrd memory: 26944K worker2 # [ 1.036467] tun: Universal TUN/TAP device driver, 1.6 worker2 # [ 1.041565] thunder_xcv, ver 1.0 worker2 # [ 1.041620] thunder_bgx, ver 1.0 worker2 # [ 1.041642] nicpf, ver 1.0 worker2 # [ 1.042252] e1000: Intel(R) PRO/1000 Network Driver worker2 # [ 1.042265] e1000: Copyright (c) 1999-2006 Intel Corporation. worker1 # [ 0.976861] Freeing initrd memory: 26976K worker2 # [ 1.042290] e1000e: Intel(R) PRO/1000 Network Driver worker2 # [ 1.042299] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker2 # [ 1.042331] igb: Intel(R) Gigabit Ethernet Network Driver worker2 # [ 1.042337] igb: Copyright (c) 2007-2014 Intel Corporation. worker2 # [ 1.042386] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker2 # [ 1.042392] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker2 # [ 1.042533] sky2: driver version 1.30 worker2 # [ 1.044151] usbcore: registered new interface driver usb-storage worker2 # [ 1.044216] usbcore: registered new interface driver usbserial_generic worker1 # [ 0.983258] tun: Universal TUN/TAP device driver, 1.6 worker2 # [ 1.044230] usbserial: USB Serial support registered for generic worker2 # [ 1.044846] hv_vmbus: registering driver hyperv_keyboard worker2 # [ 1.045910] ehci-pci 0000:00:07.0: EHCI Host Controller worker2 # [ 1.045946] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker2 # [ 1.046241] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker2 # [ 1.059467] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker2 # [ 1.059899] hub 1-0:1.0: USB hub found worker2 # [ 1.059929] hub 1-0:1.0: 6 ports detected worker1 # [ 0.987420] thunder_xcv, ver 1.0 worker1 # [ 0.987466] thunder_bgx, ver 1.0 worker1 # [ 0.987492] nicpf, ver 1.0 worker1 # [ 0.988087] e1000: Intel(R) PRO/1000 Network Driver worker2 # [ 1.063731] rtc-pl031 9010000.pl031: registered as rtc0 worker1 # [ 0.988095] e1000: Copyright (c) 1999-2006 Intel Corporation. worker2 # [ 1.063774] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T05:12:13 UTC (1790140333) worker1 # [ 0.988120] e1000e: Intel(R) PRO/1000 Network Driver worker2 # [ 1.064120] i2c_dev: i2c /dev entries driver worker1 # [ 0.988128] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker1 # [ 0.988157] igb: Intel(R) Gigabit Ethernet Network Driver worker1 # [ 0.988163] igb: Copyright (c) 2007-2014 Intel Corporation. worker1 # [ 0.988186] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker1 # [ 0.988192] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker1 # [ 0.988324] sky2: driver version 1.30 worker1 # [ 0.989962] usbcore: registered new interface driver usb-storage worker1 # [ 0.990025] usbcore: registered new interface driver usbserial_generic worker1 # [ 0.990039] usbserial: USB Serial support registered for generic worker2 # [ 1.069973] sdhci: Secure Digital Host Controller Interface driver worker1 # [ 0.990655] hv_vmbus: registering driver hyperv_keyboard worker2 # [ 1.069992] sdhci: Copyright(c) Pierre Ossman worker1 # [ 0.991611] ehci-pci 0000:00:07.0: EHCI Host Controller worker2 # [ 1.070266] Synopsys Designware Multimedia Card Interface Driver worker1 # [ 0.991639] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker2 # [ 1.070644] sdhci-pltfm: SDHCI platform and OF driver helper worker1 # [ 0.991814] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker2 # [ 1.072334] hid: raw HID events driver (C) Jiri Kosina worker2 # [ 1.072585] usbcore: registered new interface driver usbhid worker2 # [ 1.072592] usbhid: USB HID core driver worker1 # [ 1.004681] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker1 # [ 1.005059] hub 1-0:1.0: USB hub found worker1 # [ 1.005080] hub 1-0:1.0: 6 ports detected lb # [ 0.977407] Freeing initrd memory: 26948K worker1 # [ 1.008124] rtc-pl031 9010000.pl031: registered as rtc0 worker2 # [ 1.078678] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker2 # [ 1.080330] drop_monitor: Initializing network drop monitor service worker1 # [ 1.008159] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T05:12:13 UTC (1790140333) worker2 # [ 1.080513] NET: Registered PF_INET6 protocol family worker1 # [ 1.008468] i2c_dev: i2c /dev entries driver worker2 # [ 1.084130] Segment Routing with IPv6 worker2 # [ 1.084175] In-situ OAM (IOAM) with IPv6 worker2 # [ 1.084206] NET: Registered PF_PACKET protocol family lb # [ 0.984024] tun: Universal TUN/TAP device driver, 1.6 worker1 # [ 1.014056] sdhci: Secure Digital Host Controller Interface driver worker1 # [ 1.014073] sdhci: Copyright(c) Pierre Ossman worker1 # [ 1.014340] Synopsys Designware Multimedia Card Interface Driver worker1 # [ 1.014721] sdhci-pltfm: SDHCI platform and OF driver helper worker2 # [ 1.086021] 9pnet: Installing 9P2000 support worker2 # [ 1.086128] Key type dns_resolver registered lb # [ 0.987903] thunder_xcv, ver 1.0 lb # [ 0.987952] thunder_bgx, ver 1.0 lb # [ 0.987976] nicpf, ver 1.0 lb # [ 0.988609] e1000: Intel(R) PRO/1000 Network Driver worker1 # [ 1.019230] hid: raw HID events driver (C) Jiri Kosina lb # [ 0.988619] e1000: Copyright (c) 1999-2006 Intel Corporation. worker1 # [ 1.019484] usbcore: registered new interface driver usbhid lb # [ 0.988661] e1000e: Intel(R) PRO/1000 Network Driver worker1 # [ 1.019491] usbhid: USB HID core driver lb # [ 0.988668] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. lb # [ 0.988701] igb: Intel(R) Gigabit Ethernet Network Driver worker1 # [ 1.022509] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available lb # [ 0.988707] igb: Copyright (c) 2007-2014 Intel Corporation. worker2 # [ 1.094937] registered taskstats version 1 worker2 # [ 1.095109] Loading compiled-in X.509 certificates worker1 # [ 1.025246] drop_monitor: Initializing network drop monitor service client # [ 1.047511] Freeing initrd memory: 26972K worker1 # [ 1.025466] NET: Registered PF_INET6 protocol family worker1 # [ 1.027755] Segment Routing with IPv6 lb # [ 0.994129] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker1 # [ 1.027779] In-situ OAM (IOAM) with IPv6 lb # [ 0.994139] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker1 # [ 1.027830] NET: Registered PF_PACKET protocol family lb # [ 0.994292] sky2: driver version 1.30 lb # [ 0.996109] usbcore: registered new interface driver usb-storage lb # [ 0.996208] usbcore: registered new interface driver usbserial_generic worker1 # [ 1.029599] 9pnet: Installing 9P2000 support worker1 # [ 1.029650] Key type dns_resolver registered client # [ 1.054283] tun: Universal TUN/TAP device driver, 1.6 lb # [ 0.996225] usbserial: USB Serial support registered for generic lb # [ 0.997503] ehci-pci 0000:00:07.0: EHCI Host Controller worker2 # [ 1.104270] Demotion targets for Node 0: null lb # [ 0.997530] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker2 # [ 1.104432] Key type .fscrypt registered lb # [ 0.997704] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker2 # [ 1.104448] Key type fscrypt-provisioning registered lb # [ 1.004439] hv_vmbus: registering driver hyperv_keyboard worker2 # [ 1.104570] ima: No TPM chip found, activating TPM-bypass! worker2 # [ 1.104592] ima: Allocated hash algorithm: sha1 worker2 # [ 1.104618] ima: No architecture policies found lb # [ 1.006169] rtc-pl031 9010000.pl031: registered as rtc0 worker2 # [ 1.109233] input: gpio-keys as /devices/platform/gpio-keys/input/input0 lb # [ 1.006201] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T05:12:13 UTC (1790140333) worker1 # [ 1.036597] registered taskstats version 1 lb # [ 1.006523] i2c_dev: i2c /dev entries driver worker1 # [ 1.036767] Loading compiled-in X.509 certificates client # [ 1.058575] thunder_xcv, ver 1.0 lb # [ 1.009769] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 client # [ 1.058621] thunder_bgx, ver 1.0 client # [ 1.058648] nicpf, ver 1.0 lb # [ 1.010814] hub 1-0:1.0: USB hub found lb # [ 1.011359] hub 1-0:1.0: 6 ports detected client # [ 1.059233] e1000: Intel(R) PRO/1000 Network Driver client # [ 1.059242] e1000: Copyright (c) 1999-2006 Intel Corporation. client # [ 1.059266] e1000e: Intel(R) PRO/1000 Network Driver client # [ 1.059276] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. client # [ 1.059310] igb: Intel(R) Gigabit Ethernet Network Driver client # [ 1.059317] igb: Copyright (c) 2007-2014 Intel Corporation. client # [ 1.059341] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker1 # [ 1.045703] Demotion targets for Node 0: null client # [ 1.059347] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker1 # [ 1.045819] Key type .fscrypt registered lb # [ 1.014813] sdhci: Secure Digital Host Controller Interface driver client # [ 1.059488] sky2: driver version 1.30 worker1 # [ 1.045826] Key type fscrypt-provisioning registered lb # [ 1.014829] sdhci: Copyright(c) Pierre Ossman client # [ 1.061152] usbcore: registered new interface driver usb-storage worker1 # [ 1.045924] ima: No TPM chip found, activating TPM-bypass! lb # [ 1.015152] Synopsys Designware Multimedia Card Interface Driver client # [ 1.061219] usbcore: registered new interface driver usbserial_generic worker1 # [ 1.045944] ima: Allocated hash algorithm: sha1 lb # [ 1.015541] sdhci-pltfm: SDHCI platform and OF driver helper worker1 # [ 1.045967] ima: No architecture policies found client # [ 1.061234] usbserial: USB Serial support registered for generic client # [ 1.061866] hv_vmbus: registering driver hyperv_keyboard worker1 # [ 1.050368] input: gpio-keys as /devices/platform/gpio-keys/input/input0 lb # [ 1.019967] hid: raw HID events driver (C) Jiri Kosina client # [ 1.062831] ehci-pci 0000:00:07.0: EHCI Host Controller lb # [ 1.020247] usbcore: registered new interface driver usbhid client # [ 1.062860] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 lb # [ 1.020254] usbhid: USB HID core driver client # [ 1.063100] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 client # [ 1.076216] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 client # [ 1.076599] hub 1-0:1.0: USB hub found client # [ 1.076622] hub 1-0:1.0: 6 ports detected lb # [ 1.023280] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available client # [ 1.079828] rtc-pl031 9010000.pl031: registered as rtc0 lb # [ 1.027529] drop_monitor: Initializing network drop monitor service lb # [ 1.027728] NET: Registered PF_INET6 protocol family client # [ 1.079866] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T05:12:13 UTC (1790140333) lb # [ 1.029804] Segment Routing with IPv6 client # [ 1.080254] i2c_dev: i2c /dev entries driver lb # [ 1.029824] In-situ OAM (IOAM) with IPv6 lb # [ 1.029854] NET: Registered PF_PACKET protocol family lb # [ 1.031532] 9pnet: Installing 9P2000 support lb # [ 1.031586] Key type dns_resolver registered client # [ 1.086124] sdhci: Secure Digital Host Controller Interface driver client # [ 1.086135] sdhci: Copyright(c) Pierre Ossman client # [ 1.086406] Synopsys Designware Multimedia Card Interface Driver client # [ 1.086788] sdhci-pltfm: SDHCI platform and OF driver helper client # [ 1.088449] hid: raw HID events driver (C) Jiri Kosina client # [ 1.088729] usbcore: registered new interface driver usbhid worker1 # [ 1.070253] clk: Disabling unused clocks lb # [ 1.038636] registered taskstats version 1 client # [ 1.088737] usbhid: USB HID core driver lb # [ 1.038785] Loading compiled-in X.509 certificates worker1 # [ 1.070292] PM: genpd: Disabling unused power domains worker2 # [ 1.141964] clk: Disabling unused clocks worker2 # [ 1.142006] PM: genpd: Disabling unused power domains client # [ 1.094672] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker1 # [ 1.074637] Freeing unused kernel memory: 4736K client # [ 1.096354] drop_monitor: Initializing network drop monitor service worker2 # [ 1.146449] Freeing unused kernel memory: 4736K worker1 # [ 1.074846] Run /init as init process worker2 # [ 1.146693] Run /init as init process client # [ 1.096526] NET: Registered PF_INET6 protocol family client # [ 1.099984] Segment Routing with IPv6 client # [ 1.100034] In-situ OAM (IOAM) with IPv6 client # [ 1.100070] NET: Registered PF_PACKET protocol family lb # [ 1.047847] Demotion targets for Node 0: null client # [ 1.101764] 9pnet: Installing 9P2000 support lb # [ 1.047979] Key type .fscrypt registered client # [ 1.101828] Key type dns_resolver registered lb # [ 1.047986] Key type fscrypt-provisioning registered lb # [ 1.048099] ima: No TPM chip found, activating TPM-bypass! lb # [ 1.048123] ima: Allocated hash algorithm: sha1 lb # [ 1.048146] ima: No architecture policies found lb # [ 1.052423] input: gpio-keys as /devices/platform/gpio-keys/input/input0 client # [ 1.108822] registered taskstats version 1 client # [ 1.108997] Loading compiled-in X.509 certificates worker2 # [ 1.164236] systemd[1]: Successfully made /usr/ read-only. worker1 # [ 1.095325] systemd[1]: Successfully made /usr/ read-only. client # [ 1.118195] Demotion targets for Node 0: null client # [ 1.118339] Key type .fscrypt registered client # [ 1.118348] Key type fscrypt-provisioning registered client # [ 1.118469] ima: No TPM chip found, activating TPM-bypass! client # [ 1.118491] ima: Allocated hash algorithm: sha1 client # [ 1.118518] ima: No architecture policies found client # [ 1.122900] input: gpio-keys as /devices/platform/gpio-keys/input/input0 lb # [ 1.070785] clk: Disabling unused clocks lb # [ 1.071154] PM: genpd: Disabling unused power domains lb # [ 1.075680] Freeing unused kernel memory: 4736K lb # [ 1.075872] Run /init as init process lb # [ 1.091973] systemd[1]: Successfully made /usr/ read-only. client # [ 1.156509] clk: Disabling unused clocks client # [ 1.156545] PM: genpd: Disabling unused power domains client # [ 1.160895] Freeing unused kernel memory: 4736K client # [ 1.161103] Run /init as init process client # [ 1.177114] systemd[1]: Successfully made /usr/ read-only. worker2 # [ 1.305190] usb 1-1: new high-speed USB device number 2 using ehci-pci worker1 # [ 1.250973] usb 1-1: new high-speed USB device number 2 using ehci-pci lb # [ 1.257654] usb 1-1: new high-speed USB device number 2 using ehci-pci client # [ 1.322074] usb 1-1: new high-speed USB device number 2 using ehci-pci worker2 # [ 1.458336] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker1 # [ 1.409813] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker2 # [ 1.499340] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 1.503481] systemd[1]: Detected virtualization qemu. worker2 # [ 1.504181] systemd[1]: Detected architecture arm64. worker2 # [ 1.504855] systemd[1]: Running in initrd. worker2 # [ 1.505922] systemd[1]: Initializing machine ID from random generator. worker2 # [ 1.506937] systemd[1]: Hostname set to . worker1 # [ 1.430498] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker1 # [ 1.443418] systemd[1]: Detected virtualization qemu. worker1 # [ 1.445594] systemd[1]: Detected architecture arm64. worker1 # [ 1.447690] systemd[1]: Running in initrd. lb # [ 1.413633] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker1 # [ 1.450741] systemd[1]: Initializing machine ID from random generator. worker1 # [ 1.454032] systemd[1]: Hostname set to . client # [ 1.474782] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 lb # [ 1.427387] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 1.440301] systemd[1]: Detected virtualization qemu. lb # [ 1.440410] systemd[1]: Detected architecture arm64. lb # [ 1.440435] systemd[1]: Running in initrd. lb # [ 1.447516] systemd[1]: Initializing machine ID from random generator. lb # [ 1.450650] systemd[1]: Hostname set to . worker2 # [ 1.556802] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 client # [ 1.512254] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 1.525006] systemd[1]: Detected virtualization qemu. client # [ 1.527285] systemd[1]: Detected architecture arm64. client # [ 1.529315] systemd[1]: Running in initrd. worker1 # [ 1.507235] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 client # [ 1.534344] systemd[1]: Initializing machine ID from random generator. client # [ 1.537679] systemd[1]: Hostname set to . client # [ 1.554448] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 lb # [ 1.509065] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker2 # [ 1.677105] usb 1-2: new high-speed USB device number 3 using ehci-pci worker1 # [ 1.630940] usb 1-2: new high-speed USB device number 3 using ehci-pci client # [ 1.686086] usb 1-2: new high-speed USB device number 3 using ehci-pci lb # [ 1.632786] usb 1-2: new high-speed USB device number 3 using ehci-pci worker2 # [ 1.777550] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 1.829579] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.833339] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker1 # [ 1.789694] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 1.793704] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.874311] systemd[1]: Queued start job for default target Initrd Default Target. worker1 # [ 1.799450] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker2 # [ 1.883526] systemd[1]: Created slice Slice /system/modprobe. worker2 # [ 1.884659] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker2 # [ 1.885972] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker2 # [ 1.887029] systemd[1]: Reached target Path Units. worker2 # [ 1.887772] systemd[1]: Reached target Slice Units. worker2 # [ 1.888533] systemd[1]: Reached target Swaps. worker2 # [ 1.889274] systemd[1]: Reached target Timer Units. worker2 # [ 1.890218] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 1.891345] systemd[1]: Listening on Journal Socket (/dev/log). lb # [ 1.785145] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.892417] systemd[1]: Listening on Journal Sockets. client # [ 1.838725] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.893374] systemd[1]: Listening on udev Control Socket. worker2 # [ 1.894306] systemd[1]: Listening on udev Kernel Socket. worker2 # [ 1.895225] systemd[1]: Reached target Socket Units. lb # [ 1.790844] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker2 # [ 1.897648] systemd[1]: Starting Create List of Static Device Nodes... client # [ 1.846289] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker2 # [ 1.898746] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 1.907260] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 1.917291] systemd[1]: Starting Journal Service... client # [ 1.877375] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 1.926287] systemd[1]: Starting Load Kernel Modules... worker2 # [ 1.928757] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 1.957347] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 1.895701] systemd[1]: Queued start job for default target Initrd Default Target. worker2 # [ 1.973619] systemd[1]: Finished Create List of Static Device Nodes. worker1 # [ 1.905406] systemd[1]: Created slice Slice /system/modprobe. worker1 # [ 1.906708] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 1.908185] systemd[1]: Expecting device /dev/disk/by-label/nixos... lb # [ 1.880558] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 1.908223] systemd[1]: Reached target Path Units. worker1 # [ 1.908244] systemd[1]: Reached target Slice Units. worker1 # [ 1.908266] systemd[1]: Reached target Swaps. worker2 # [ 1.985502] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker1 # [ 1.908286] systemd[1]: Reached target Timer Units. worker2 # [ 1.986903] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 1.908482] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 1.908680] systemd[1]: Listening on Journal Socket (/dev/log). worker1 # [ 1.908874] systemd[1]: Listening on Journal Sockets. worker1 # [ 1.909018] systemd[1]: Listening on udev Control Socket. worker1 # [ 1.909156] systemd[1]: Listening on udev Kernel Socket. worker1 # [ 1.909181] systemd[1]: Reached target Socket Units. worker1 # [ 1.921642] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 1.995083] systemd-journald[72]: Collecting audit messages is disabled. worker1 # [ 1.922829] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 1.934646] systemd[1]: Mounting Kernel Configuration File System... client # [ 1.977268] systemd[1]: Queued start job for default target Initrd Default Target. worker1 # [ 1.963433] systemd[1]: Starting Journal Service... client # [ 1.988721] systemd[1]: Created slice Slice /system/modprobe. worker1 # [ 1.966187] systemd[1]: Starting Load Kernel Modules... client # [ 1.989971] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 1.967286] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 1.991391] systemd[1]: Expecting device /dev/disk/by-label/nixos... client # [ 1.992537] systemd[1]: Reached target Path Units. client # [ 1.993363] systemd[1]: Reached target Slice Units. client # [ 1.994234] systemd[1]: Reached target Swaps. client # [ 1.994976] systemd[1]: Reached target Timer Units. client # [ 1.996012] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 1.997270] systemd[1]: Listening on Journal Socket (/dev/log). client # [ 1.998510] systemd[1]: Listening on Journal Sockets. client # [ 1.998682] systemd[1]: Listening on udev Control Socket. worker1 # [ 1.978609] systemd[1]: Starting Coldplug All udev Devices... client # [ 1.998811] systemd[1]: Listening on udev Kernel Socket. client # [ 1.998837] systemd[1]: Reached target Socket Units. client # [ 2.004188] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 2.050740] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. client # [ 2.005383] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 2.051716] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 2.061534] systemd[1]: Starting Create Static Device Nodes in /dev... client # [ 2.014174] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 1.992418] systemd[1]: Finished Create List of Static Device Nodes. worker1 # [ 1.993198] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 2.069166] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev client # [ 2.022336] systemd[1]: Starting Journal Service... worker1 # [ 2.011155] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 2.082534] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker2 # [ 2.082779] [drm] features: -virgl +edid -resource_blob -host_visible worker2 # [ 2.082790] [drm] features: -context_init worker2 # [ 2.083547] [drm] number of scanouts: 1 worker2 # [ 2.083569] [drm] number of cap sets: 0 client # [ 2.046643] systemd[1]: Starting Load Kernel Modules... lb # [ 1.994431] systemd[1]: Queued start job for default target Initrd Default Target. client # [ 2.046775] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 2.101843] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic lb # [ 2.003690] systemd[1]: Created slice Slice /system/modprobe. worker2 # [ 2.101870] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 lb # [ 2.005142] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 2.006647] systemd[1]: Expecting device /dev/disk/by-label/nixos... lb # [ 2.007878] systemd[1]: Reached target Path Units. lb # [ 2.008883] systemd[1]: Reached target Slice Units. lb # [ 2.009845] systemd[1]: Reached target Swaps. lb # [ 2.010713] systemd[1]: Reached target Timer Units. worker1 # [ 2.039793] systemd-journald[72]: Collecting audit messages is disabled. lb # [ 2.011917] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 2.013439] systemd[1]: Listening on Journal Socket (/dev/log). lb # [ 2.014741] systemd[1]: Listening on Journal Sockets. lb # [ 2.015965] systemd[1]: Listening on udev Control Socket. client # [ 2.069745] systemd-journald[72]: Collecting audit messages is disabled. lb # [ 2.017205] systemd[1]: Listening on udev Kernel Socket. lb # [ 2.017243] systemd[1]: Reached target Socket Units. lb # [ 2.021048] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 2.048223] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. client # [ 2.074187] systemd[1]: Starting Coldplug All udev Devices... lb # [ 2.022337] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 2.125362] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 2.125600] systemd[1]: Reached target Preparation for Local File Systems. client # [ 2.078997] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 2.125627] systemd[1]: Reached target Local File Systems. client # [ 2.080629] systemd[1]: Mounted Kernel Configuration File System. lb # [ 2.033016] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 2.063667] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 2.066468] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 2.137600] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 2.040099] systemd[1]: Starting Journal Service... client # [ 2.095263] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker1 # [ 2.071020] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker2 # [ 2.142381] Console: switching to colour frame buffer device 160x50 worker1 # [ 2.077589] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker1 # [ 2.077842] [drm] features: -virgl +edid -resource_blob -host_visible worker1 # [ 2.077853] [drm] features: -context_init worker1 # [ 2.078619] [drm] number of scanouts: 1 worker1 # [ 2.078638] [drm] number of cap sets: 0 lb # [ 2.058078] systemd[1]: Starting Load Kernel Modules... lb # [ 2.058230] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 2.163291] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device lb # [ 2.073903] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 2.181810] systemd[1]: Finished Load Kernel Modules. worker1 # [ 2.107654] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker1 # [ 2.107681] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 worker2 # [ 2.189587] systemd[1]: Starting Apply Kernel Variables... lb # [ 2.096940] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 2.097808] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 2.185199] systemd-modules-load[73]: Inserted module 'dm_mod' worker2 # [ 2.186287] systemd-modules-load[73]: Module 'virtio_balloon' is built in worker2 # [ 2.187384] systemd-modules-load[73]: Module 'virtio_console' is built in client # [ 2.154773] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 2.131690] systemd[1]: Finished Create Static Device Nodes in /dev. worker1 # [ 2.132091] systemd[1]: Reached target Preparation for Local File Systems. client # [ 2.157565] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 2.132119] systemd[1]: Reached target Local File Systems. lb # [ 2.105167] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 2.209302] systemd[1]: Started Journal Service. worker2 # [ 2.196476] systemd-modules-load[73]: Inserted module 'virtio_gpu' client # [ 2.167948] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker1 # [ 2.139193] Console: switching to colour frame buffer device 160x50 worker2 # [ 2.204354] systemd-modules-load[73]: Module 'virtio_rng' is built in worker1 # [ 2.149919] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 2.136009] systemd-journald[72]: Collecting audit messages is disabled. worker1 # [ 2.171466] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device client # [ 2.186046] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker2 # [ 2.230608] systemd[1]: Starting Create System Files and Directories... client # [ 2.192687] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 client # [ 2.192955] [drm] features: -virgl +edid -resource_blob -host_visible client # [ 2.192964] [drm] features: -context_init client # [ 2.193936] [drm] number of scanouts: 1 client # [ 2.193955] [drm] number of cap sets: 0 lb # [ 2.154773] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker1 # [ 2.187528] systemd[1]: Finished Load Kernel Modules. worker2 # [ 2.248523] systemd-udevd[79]: Using default interface naming scheme 'v261'. worker1 # [ 2.195452] systemd[1]: Starting Apply Kernel Variables... client # [ 2.223366] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 2.223761] systemd[1]: Reached target Preparation for Local File Systems. client # [ 2.223793] systemd[1]: Reached target Local File Systems. worker2 # [ 2.261982] systemd[1]: Finished Apply Kernel Variables. client # [ 2.228088] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic client # [ 2.228108] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 lb # [ 2.176875] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev client # [ 2.236266] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 2.183618] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 lb # [ 2.183902] [drm] features: -virgl +edid -resource_blob -host_visible lb # [ 2.183918] [drm] features: -context_init worker2 # [ 2.278110] systemd[1]: Finished Create System Files and Directories. lb # [ 2.200802] [drm] number of scanouts: 1 lb # [ 2.200840] [drm] number of cap sets: 0 lb # [ 2.202781] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 2.237242] systemd[1]: Started Journal Service. client # [ 2.255043] Console: switching to colour frame buffer device 160x50 lb # [ 2.205829] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker2 # [ 2.297173] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 2.205849] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 worker1 # [ 2.232358] systemd-modules-load[73]: Inserted module 'dm_mod' client # [ 2.252656] systemd-modules-load[74]: Inserted module 'dm_mod' client # [ 2.256263] systemd-modules-load[74]: Module 'virtio_balloon' is built in worker1 # [ 2.236537] systemd-modules-load[73]: Module 'virtio_balloon' is built in worker1 # [ 2.237664] systemd-modules-load[73]: Module 'virtio_console' is built in client # [ 2.257606] systemd-modules-load[74]: Module 'virtio_console' is built in worker1 # [ 2.238776] systemd-modules-load[73]: Inserted module 'virtio_gpu' worker1 # [ 2.239780] systemd-modules-load[73]: Module 'virtio_rng' is built in client # [ 2.277306] systemd[1]: Started Journal Service. lb # [ 2.214752] systemd[1]: Starting Create Static Device Nodes in /dev... client # [ 2.290061] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker1 # [ 2.256595] systemd[1]: Starting Create System Files and Directories... worker1 # [ 2.257946] systemd-udevd[78]: Using default interface naming scheme 'v261'. lb # [ 2.237054] Console: switching to colour frame buffer device 160x50 client # [ 2.280867] systemd-modules-load[74]: Inserted module 'virtio_gpu' lb # [ 2.244003] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker1 # [ 2.264153] systemd[1]: Finished Apply Kernel Variables. client # [ 2.288264] systemd-modules-load[74]: Module 'virtio_rng' is built in client # [ 2.292190] systemd[1]: Starting Create System Files and Directories... client # [ 2.300794] systemd[1]: Finished Load Kernel Modules. lb # [ 2.266051] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 2.266570] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 2.266617] systemd[1]: Reached target Local File Systems. worker2 # [ 2.360988] systemd[1]: Starting Virtual Console Setup... worker1 # [ 2.293239] systemd[1]: Finished Create System Files and Directories. client # [ 2.320461] systemd-udevd[78]: Using default interface naming scheme 'v261'. client # [ 2.324198] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 2.309086] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 2.292898] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 2.295912] systemd[1]: Finished Load Kernel Modules. lb # [ 2.299821] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 2.412542] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 2.416460] systemd[1]: Finished Virtual Console Setup. client # [ 2.369886] systemd[1]: Finished Create System Files and Directories. client # [ 2.376319] systemd[1]: Finished Apply Kernel Variables. lb # [ 2.340530] systemd[1]: Started Journal Service. client # [ 2.390569] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 2.341055] systemd-modules-load[73]: Inserted module 'dm_mod' worker1 # [ 2.373267] systemd[1]: Starting Virtual Console Setup... lb # [ 2.345023] systemd-modules-load[73]: Module 'virtio_balloon' is built in lb # [ 2.346246] systemd-modules-load[73]: Module 'virtio_console' is built in lb # [ 2.347396] systemd-modules-load[73]: Inserted module 'virtio_gpu' lb # [ 2.362727] systemd-modules-load[73]: Module 'virtio_rng' is built in lb # [ 2.372595] systemd[1]: Starting Create System Files and Directories... lb # [ 2.376679] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 2.436508] systemd-vconsole-setup[100]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 2.440243] systemd[1]: Finished Virtual Console Setup. client # [ 2.460510] systemd[1]: Starting Virtual Console Setup... lb # [ 2.420487] systemd-udevd[78]: Using default interface naming scheme 'v261'. lb # [ 2.424401] systemd[1]: Finished Create System Files and Directories. lb # [ 2.453300] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 2.512533] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 2.516407] systemd[1]: Finished Virtual Console Setup. lb # [ 2.516642] systemd[1]: Starting Virtual Console Setup... lb # [ 2.572558] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 2.576138] systemd[1]: Finished Virtual Console Setup. worker2 # [ 3.035366] systemd[1]: Finished Coldplug All udev Devices. worker2 # [ 3.037079] systemd[1]: Reached target System Initialization. worker2 # [ 3.040116] systemd[1]: Reached target Basic System. worker1 # [ 3.061161] systemd[1]: Finished Coldplug All udev Devices. worker1 # [ 3.062150] systemd[1]: Reached target System Initialization. worker1 # [ 3.064101] systemd[1]: Reached target Basic System. worker2 # [ 3.172119] (udev-worker)[91]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 3.133062] systemd[1]: Finished Coldplug All udev Devices. client # [ 3.134715] systemd[1]: Reached target System Initialization. client # [ 3.140151] systemd[1]: Reached target Basic System. worker2 # [ 3.213618] (udev-worker)[104]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 3.216567] (udev-worker)[91]: Network interface NamePolicy= disabled on kernel command line. lb # [ 3.162301] systemd[1]: Finished Coldplug All udev Devices. lb # [ 3.163295] systemd[1]: Reached target System Initialization. lb # [ 3.164203] systemd[1]: Reached target Basic System. worker2 # [ 3.288622] systemd[1]: Found device /dev/disk/by-label/nixos. worker2 # [ 3.301675] systemd[1]: Reached target Initrd Root Device. worker2 # [ 3.305763] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker1 # [ 3.245282] (udev-worker)[103]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 3.249846] (udev-worker)[103]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.257959] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 3.353635] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks client # [ 3.305738] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 3.364155] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker2 # [ 3.366328] systemd[1]: Mounting /sysroot... client # [ 3.322314] (udev-worker)[96]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 3.326227] (udev-worker)[96]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 3.418945] EXT4-fs (vda): mounted filesystem 203ff707-c759-4c3a-a3f9-d5fbb101f69b r/w with ordered data mode. Quota mode: none. worker2 # [ 3.407641] systemd[1]: Mounted /sysroot. worker2 # [ 3.410021] systemd[1]: Reached target Initrd Root File System. worker1 # [ 3.341041] systemd[1]: Found device /dev/disk/by-label/nixos. worker1 # [ 3.342962] systemd[1]: Reached target Initrd Root Device. worker2 # [ 3.415359] systemd[1]: Starting Mountpoints Configured in the Real Root... worker1 # [ 3.349378] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... lb # [ 3.317506] (udev-worker)[94]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. lb # [ 3.324418] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line. client # [ 3.393957] systemd[1]: Found device /dev/disk/by-label/nixos. worker2 # [ 3.443778] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. client # [ 3.400190] systemd[1]: Reached target Initrd Root Device. worker2 # [ 3.450851] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... lb # [ 3.352823] (udev-worker)[94]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 3.455753] systemd[1]: Reloading... client # [ 3.405501] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker1 # [ 3.397312] systemd-fsck[109]: nixos: clean, 12/65536 files, 13019/262144 blocks worker1 # [ 3.404186] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker1 # [ 3.412321] systemd[1]: Mounting /sysroot... client # [ 3.457275] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks client # [ 3.464114] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. client # [ 3.468132] systemd[1]: Mounting /sysroot... worker1 # [ 3.464904] EXT4-fs (vda): mounted filesystem 6fdfa4c4-3465-40d7-a9c6-192ad8c81d98 r/w with ordered data mode. Quota mode: none. worker1 # [ 3.455626] systemd[1]: Mounted /sysroot. worker1 # [ 3.460153] systemd[1]: Reached target Initrd Root File System. worker1 # [ 3.462076] systemd[1]: Starting Mountpoints Configured in the Real Root... lb # [ 3.454559] systemd[1]: Found device /dev/disk/by-label/nixos. lb # [ 3.457707] systemd[1]: Reached target Initrd Root Device. client # [ 3.526662] EXT4-fs (vda): mounted filesystem 17878813-a2c4-493a-b0e7-fd3fa6804a97 r/w with ordered data mode. Quota mode: none. lb # [ 3.461395] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... client # [ 3.514747] systemd[1]: Mounted /sysroot. worker1 # [ 3.494308] systemd-sysroot-fstab-check[117]: /sysroot should be mounted in the initrd, will request daemon-reload. client # [ 3.517398] systemd[1]: Reached target Initrd Root File System. worker1 # [ 3.500224] systemd[1]: Reload requested from client PID 117 ('systemd-sysroot') (unit initrd-parse-etc.service)... client # [ 3.522361] systemd[1]: Starting Mountpoints Configured in the Real Root... worker1 # [ 3.504556] systemd[1]: Reloading... client # [ 3.551958] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. lb # [ 3.506574] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks client # [ 3.561063] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... lb # [ 3.512776] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. client # [ 3.565576] systemd[1]: Reloading... lb # [ 3.520968] systemd[1]: Mounting /sysroot... lb # [ 3.575478] EXT4-fs (vda): mounted filesystem 98fd74e6-f0d8-4597-b85e-8a4553334efc r/w with ordered data mode. Quota mode: none. lb # [ 3.565238] systemd[1]: Mounted /sysroot. lb # [ 3.567152] systemd[1]: Reached target Initrd Root File System. worker2 # [ 3.673420] systemd[1]: Reloading finished in 220 ms. lb # [ 3.572983] systemd[1]: Starting Mountpoints Configured in the Real Root... lb # [ 3.600226] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. worker2 # [ 3.705248] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... lb # [ 3.606964] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... worker2 # [ 3.709532] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... lb # [ 3.611795] systemd[1]: Reloading... worker2 # [ 3.715315] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker2 # [ 3.718190] systemd[1]: Finished Mountpoints Configured in the Real Root. worker2 # [ 3.721187] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker1 # [ 3.734875] systemd[1]: Reloading finished in 231 ms. client # [ 3.779616] systemd[1]: Reloading finished in 219 ms. worker1 # [ 3.762588] systemd-sysroot-fstab-check[117]: Requesting initrd-fs.target/start/replace... worker1 # [ 3.765172] systemd-sysroot-fstab-check[117]: Requesting swap.target/start/replace... worker1 # [ 3.774138] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker1 # [ 3.775256] systemd[1]: Finished Mountpoints Configured in the Real Root. worker1 # [ 3.776991] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. client # [ 3.812164] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... client # [ 3.816346] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... client # [ 3.822830] systemd[1]: initrd-parse-etc.service: Deactivated successfully. client # [ 3.826043] systemd[1]: Finished Mountpoints Configured in the Real Root. client # [ 3.827856] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. lb # [ 3.842432] systemd[1]: Reloading finished in 233 ms. worker2 # [ 3.948274] systemd[1]: Mounting /sysroot/nix/.ro-store... worker2 # [ 3.961516] systemd[1]: Mounting /sysroot/nix/.rw-store... worker2 # [ 3.973933] systemd[1]: Mounting /sysroot/run... lb # [ 3.872106] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... lb # [ 3.876671] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... lb # [ 3.884228] systemd[1]: initrd-parse-etc.service: Deactivated successfully. lb # [ 3.886774] systemd[1]: Finished Mountpoints Configured in the Real Root. lb # [ 3.889030] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker2 # [ 3.994082] systemd[1]: Mounting /sysroot/tmp/shared... worker2 # [ 4.024102] systemd[1]: Mounting /sysroot/tmp/xchg... worker2 # [ 4.055863] fuse: init (API version 7.45) worker1 # [ 3.984494] systemd[1]: Mounting /sysroot/nix/.ro-store... worker2 # [ 4.072055] virtiofs virtio6: discovered new tag: nix-store worker2 # [ 4.072856] virtiofs virtio6: virtio_fs_setup_dax: No cache capability worker2 # [ 4.086754] virtiofs virtio7: discovered new tag: shared worker1 # [ 4.002389] systemd[1]: Mounting /sysroot/nix/.rw-store... worker2 # [ 4.087564] virtiofs virtio7: virtio_fs_setup_dax: No cache capability worker1 # [ 4.009270] systemd[1]: Mounting /sysroot/run... worker2 # [ 4.096702] virtiofs virtio8: discovered new tag: xchg worker2 # [ 4.100483] virtiofs virtio8: virtio_fs_setup_dax: No cache capability worker1 # [ 4.021806] systemd[1]: Mounting /sysroot/tmp/shared... worker2 # [ 4.104227] systemd[1]: Mounted /sysroot/nix/.ro-store. client # [ 4.061055] systemd[1]: Mounting /sysroot/nix/.ro-store... worker2 # [ 4.115854] systemd[1]: Mounted /sysroot/nix/.rw-store. worker2 # [ 4.119907] systemd[1]: Mounted /sysroot/run. worker2 # [ 4.125801] systemd[1]: Mounted /sysroot/tmp/shared. client # [ 4.078095] systemd[1]: Mounting /sysroot/nix/.rw-store... worker1 # [ 4.058394] systemd[1]: Mounting /sysroot/tmp/xchg... worker2 # [ 4.130966] systemd[1]: Mounted /sysroot/tmp/xchg. worker2 # [ 4.138878] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 4.089223] systemd[1]: Mounting /sysroot/run... worker1 # [ 4.102415] fuse: init (API version 7.45) client # [ 4.114626] systemd[1]: Mounting /sysroot/tmp/shared... worker2 # [ 4.172105] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 4.071079] systemd[1]: Mounting /sysroot/nix/.ro-store... worker2 # [ 4.173555] systemd[1]: Finished rw-sysroot-nix-store.service. worker1 # [ 4.123038] virtiofs virtio6: discovered new tag: nix-store worker1 # [ 4.123890] virtiofs virtio6: virtio_fs_setup_dax: No cache capability worker1 # [ 4.113508] systemd[1]: Mounted /sysroot/nix/.rw-store. client # [ 4.134616] systemd[1]: Mounting /sysroot/tmp/xchg... lb # [ 4.083746] systemd[1]: Mounting /sysroot/nix/.rw-store... worker1 # [ 4.140730] virtiofs virtio7: discovered new tag: shared worker1 # [ 4.141537] virtiofs virtio7: virtio_fs_setup_dax: No cache capability lb # [ 4.097207] systemd[1]: Mounting /sysroot/run... worker1 # [ 4.150593] virtiofs virtio8: discovered new tag: xchg worker1 # [ 4.155476] virtiofs virtio8: virtio_fs_setup_dax: No cache capability client # [ 4.172471] systemd[1]: Mounted /sysroot/nix/.rw-store. worker1 # [ 4.155806] systemd[1]: Mounted /sysroot/run. lb # [ 4.127430] systemd[1]: Mounting /sysroot/tmp/shared... worker1 # [ 4.161631] systemd[1]: Mounted /sysroot/tmp/shared. worker1 # [ 4.168139] systemd[1]: Mounted /sysroot/nix/.ro-store. worker1 # [ 4.174807] systemd[1]: Mounted /sysroot/tmp/xchg. client # [ 4.215644] fuse: init (API version 7.45) worker1 # [ 4.182726] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 4.221881] virtiofs virtio6: discovered new tag: nix-store client # [ 4.226886] virtiofs virtio6: virtio_fs_setup_dax: No cache capability client # [ 4.212080] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 4.167264] systemd[1]: Mounting /sysroot/tmp/xchg... client # [ 4.243856] virtiofs virtio7: discovered new tag: shared worker1 # [ 4.207606] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 4.244670] virtiofs virtio7: virtio_fs_setup_dax: No cache capability worker1 # [ 4.209334] systemd[1]: Finished rw-sysroot-nix-store.service. client # [ 4.230726] systemd[1]: Mounted /sysroot/run. lb # [ 4.184147] systemd[1]: Mounted /sysroot/nix/.rw-store. lb # [ 4.207885] fuse: init (API version 7.45) lb # [ 4.219096] virtiofs virtio6: discovered new tag: nix-store lb # [ 4.219890] virtiofs virtio6: virtio_fs_setup_dax: No cache capability lb # [ 4.212230] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 4.234526] virtiofs virtio7: discovered new tag: shared lb # [ 4.235328] virtiofs virtio7: virtio_fs_setup_dax: No cache capability client # [ 4.290960] virtiofs virtio8: discovered new tag: xchg lb # [ 4.239866] virtiofs virtio8: discovered new tag: xchg lb # [ 4.240617] virtiofs virtio8: virtio_fs_setup_dax: No cache capability client # [ 4.293312] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 4.243236] systemd[1]: Mounted /sysroot/run. client # [ 4.295181] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 4.253959] systemd[1]: Mounted /sysroot/nix/.ro-store. client # [ 4.331263] virtiofs virtio8: virtio_fs_setup_dax: No cache capability lb # [ 4.266730] systemd[1]: Mounted /sysroot/tmp/shared. lb # [ 4.282603] systemd[1]: Mounted /sysroot/tmp/xchg. client # [ 4.336671] systemd[1]: Mounted /sysroot/tmp/shared. lb # [ 4.290124] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 4.342390] systemd[1]: Mounted /sysroot/nix/.ro-store. lb # [ 4.291282] systemd[1]: Finished rw-sysroot-nix-store.service. client # [ 4.344536] systemd[1]: Mounted /sysroot/tmp/xchg. worker2 # [ 4.604785] (udev-worker)[91]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker2 # [ 4.609433] (udev-worker)[91]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory worker2 # [ 4.638625] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 4.641756] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 4.644185] systemd[1]: Stopping Virtual Console Setup... worker2 # [ 4.647022] systemd[1]: Starting Virtual Console Setup... worker2 # [ 4.664813] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 4.666294] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 4.668749] systemd[1]: Starting Virtual Console Setup... worker2 # [ 4.690714] systemd-vconsole-setup[152]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 4.694209] systemd[1]: Finished Virtual Console Setup. worker1 # [ 4.679264] (udev-worker)[96]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker1 # [ 4.684674] (udev-worker)[96]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory worker1 # [ 4.720126] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.721333] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 4.724150] systemd[1]: Stopping Virtual Console Setup... worker1 # [ 4.729208] systemd[1]: Starting Virtual Console Setup... client # [ 4.754619] (udev-worker)[92]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker1 # [ 4.736611] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.738122] systemd[1]: Stopped Virtual Console Setup. client # [ 4.760597] (udev-worker)[92]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory worker1 # [ 4.744321] systemd[1]: Starting Virtual Console Setup... client # [ 4.792078] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.793252] systemd[1]: Stopped Virtual Console Setup. client # [ 4.794748] systemd[1]: Stopping Virtual Console Setup... worker1 # [ 4.774061] systemd-vconsole-setup[152]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 4.777627] systemd[1]: Finished Virtual Console Setup. client # [ 4.799543] systemd[1]: Starting Virtual Console Setup... lb # [ 4.757285] (udev-worker)[104]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory client # [ 4.812684] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.814267] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.764940] (udev-worker)[104]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory client # [ 4.821042] systemd[1]: Starting Virtual Console Setup... lb # [ 4.796389] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.850085] systemd-vconsole-setup[156]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 4.799515] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.800424] systemd[1]: Stopping Virtual Console Setup... client # [ 4.853369] systemd[1]: Finished Virtual Console Setup. lb # [ 4.804118] systemd[1]: Starting Virtual Console Setup... lb # [ 4.837566] systemd-vconsole-setup[152]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 4.841256] systemd[1]: Finished Virtual Console Setup. worker2 # [ 4.951340] systemd[1]: Mounting /sysroot/nix/store... worker2 # [ 4.999782] systemd[1]: Mounted /sysroot/nix/store. worker2 # [ 5.001980] systemd[1]: Reached target Initrd File Systems. worker2 # [ 5.005295] systemd[1]: Starting Find NixOS closure... worker2 # [ 5.016423] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker2 # [ 5.047278] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker2 # [ 5.049743] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker2 # [ 5.057958] systemd[1]: Finished Find NixOS closure. worker1 # [ 4.987689] systemd[1]: Mounting /sysroot/nix/store... worker2 # [ 5.059566] systemd[1]: Reached target Initrd Default Target. worker2 # [ 5.062447] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker2 # [ 5.088999] systemd[1]: Stopped target Initrd Default Target. worker2 # [ 5.090329] systemd[1]: Stopped target Basic System. worker2 # [ 5.092305] systemd[1]: Stopped target Initrd Root Device. worker2 # [ 5.093529] systemd[1]: Stopped target Path Units. worker2 # [ 5.096349] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker2 # [ 5.098039] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker2 # [ 5.100126] systemd[1]: Stopped target Slice Units. worker2 # [ 5.104503] systemd[1]: Stopped target Socket Units. worker2 # [ 5.105333] systemd[1]: Stopped target System Initialization. worker2 # [ 5.106169] systemd[1]: Stopped target Swaps. worker2 # [ 5.107164] systemd[1]: Stopped target Timer Units. worker2 # [ 5.108913] systemd[1]: dbus.socket: Deactivated successfully. worker2 # [ 5.113458] systemd[1]: Closed D-Bus System Message Bus Socket. client # [ 5.064153] systemd[1]: Mounting /sysroot/nix/store... worker2 # [ 5.117994] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker2 # [ 5.121037] systemd[1]: Stopped Find NixOS closure. worker2 # [ 5.126996] systemd[1]: Starting rw-sysroot-nix-store.service... worker2 # [ 5.127957] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker2 # [ 5.130410] systemd[1]: Stopped Apply Kernel Variables. worker1 # [ 5.061537] systemd[1]: Mounted /sysroot/nix/store. worker2 # [ 5.133641] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker1 # [ 5.066651] systemd[1]: Reached target Initrd File Systems. worker1 # [ 5.069052] systemd[1]: Starting Find NixOS closure... worker2 # [ 5.140327] systemd[1]: Stopped Load Kernel Modules. worker2 # [ 5.141993] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker2 # [ 5.144351] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker1 # [ 5.076787] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker2 # [ 5.148383] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker2 # [ 5.151675] systemd[1]: Stopped Create System Files and Directories. worker2 # [ 5.156556] systemd[1]: Stopped target Local File Systems. worker2 # [ 5.157370] systemd[1]: Stopped target Preparation for Local File Systems. worker2 # [ 5.158295] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. worker2 # [ 5.159289] systemd[1]: Stopped Coldplug All udev Devices. worker2 # [ 5.161466] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker2 # [ 5.163923] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 5.165260] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 5.165995] systemd[1]: initrd-cleanup.service: Deactivated successfully. worker2 # [ 5.167135] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker2 # [ 5.168209] systemd[1]: systemd-udevd.service: Deactivated successfully. worker2 # [ 5.169153] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker2 # [ 5.170148] systemd[1]: systemd-udevd.service: Consumed 1.426s CPU time over 3.021s wall clock time, 22.4M memory peak. worker2 # [ 5.171557] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. lb # [ 5.071912] systemd[1]: Mounting /sysroot/nix/store... worker2 # [ 5.176397] systemd[1]: Closed udev Control Socket. worker2 # [ 5.178112] systemd[1]: Starting Cleanup udev Database... worker2 # [ 5.180442] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. client # [ 5.132943] systemd[1]: Mounted /sysroot/nix/store. worker2 # [ 5.184332] systemd[1]: Stopped Create Static Device Nodes in /dev. client # [ 5.136400] systemd[1]: Reached target Initrd File Systems. worker1 # [ 5.118008] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker2 # [ 5.188300] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker2 # [ 5.189435] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. client # [ 5.140913] systemd[1]: Starting Find NixOS closure... worker2 # [ 5.192096] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker1 # [ 5.121438] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker2 # [ 5.196243] systemd[1]: Stopped Create List of Static Device Nodes. worker2 # [ 5.197146] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 5.198118] systemd[1]: Finished rw-sysroot-nix-store.service. worker1 # [ 5.131660] systemd[1]: Finished Find NixOS closure. client # [ 5.152398] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker1 # [ 5.135508] systemd[1]: Reached target Initrd Default Target. worker1 # [ 5.140492] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... lb # [ 5.118235] systemd[1]: Mounted /sysroot/nix/store. lb # [ 5.120411] systemd[1]: Reached target Initrd File Systems. lb # [ 5.123226] systemd[1]: Starting Find NixOS closure... worker2 # [ 5.225404] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker2 # [ 5.228651] systemd[1]: Finished Cleanup udev Database. worker2 # [ 5.229494] systemd[1]: Reached target Switch Root. worker2 # [ 5.232471] systemd[1]: Starting NixOS Activation... lb # [ 5.132306] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker1 # [ 5.165573] systemd[1]: Stopped target Initrd Default Target. worker1 # [ 5.168197] systemd[1]: Stopped target Basic System. worker1 # [ 5.169245] systemd[1]: Stopped target Initrd Root Device. worker1 # [ 5.170548] systemd[1]: Stopped target Path Units. worker1 # [ 5.172200] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. client # [ 5.193423] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker1 # [ 5.174478] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker1 # [ 5.180157] systemd[1]: Stopped target Slice Units. worker1 # [ 5.180952] systemd[1]: Stopped target Socket Units. worker1 # [ 5.181647] systemd[1]: Stopped target System Initialization. worker1 # [ 5.182485] systemd[1]: Stopped target Swaps. worker1 # [ 5.183117] systemd[1]: Stopped target Timer Units. client # [ 5.201579] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker1 # [ 5.183805] systemd[1]: dbus.socket: Deactivated successfully. worker1 # [ 5.187451] systemd[1]: Closed D-Bus System Message Bus Socket. client # [ 5.212161] systemd[1]: Finished Find NixOS closure. worker1 # [ 5.192460] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker1 # [ 5.193552] systemd[1]: Stopped Find NixOS closure. worker1 # [ 5.194274] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 5.214291] systemd[1]: Reached target Initrd Default Target. worker1 # [ 5.195118] systemd[1]: systemd-sysctl.service: Deactivated successfully. client # [ 5.218029] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker1 # [ 5.206226] systemd[1]: Stopped Apply Kernel Variables. lb # [ 5.174110] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker1 # [ 5.207188] systemd[1]: systemd-modules-load.service: Deactivated successfully. lb # [ 5.179593] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker1 # [ 5.213154] systemd[1]: Stopped Load Kernel Modules. worker1 # [ 5.214225] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. lb # [ 5.185762] systemd[1]: Finished Find NixOS closure. worker1 # [ 5.218177] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. lb # [ 5.187527] systemd[1]: Reached target Initrd Default Target. worker1 # [ 5.219981] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. lb # [ 5.190781] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker1 # [ 5.224199] systemd[1]: Stopped Create System Files and Directories. worker1 # [ 5.226561] systemd[1]: Stopped target Local File Systems. worker1 # [ 5.228233] systemd[1]: Stopped target Preparation for Local File Systems. worker1 # [ 5.229988] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. client # [ 5.250472] systemd[1]: Stopped target Initrd Default Target. worker1 # [ 5.232286] systemd[1]: Stopped Coldplug All udev Devices. client # [ 5.252598] systemd[1]: Stopped target Basic System. client # [ 5.256152] systemd[1]: Stopped target Initrd Root Device. worker1 # [ 5.236561] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker1 # [ 5.237686] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 5.257769] systemd[1]: Stopped target Path Units. worker1 # [ 5.238723] systemd[1]: Stopped Virtual Console Setup. client # [ 5.258801] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker1 # [ 5.239445] systemd[1]: initrd-cleanup.service: Deactivated successfully. worker1 # [ 5.241477] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker1 # [ 5.242435] systemd[1]: systemd-udevd.service: Deactivated successfully. client # [ 5.262689] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker1 # [ 5.243361] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker1 # [ 5.244574] systemd[1]: systemd-udevd.service: Consumed 1.477s CPU time over 3.090s wall clock time, 22.4M memory peak. worker2 # [ 5.314622] initrd-nixos-activation-start[176]: booting system configuration /nix/store/zmhsivlz18fkqv0yqzlasss6r4v9smgp-nixos-system-worker2-test worker1 # [ 5.245984] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker1 # [ 5.246981] systemd[1]: Finished rw-sysroot-nix-store.service. worker1 # [ 5.247789] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. client # [ 5.268258] systemd[1]: Stopped target Slice Units. client # [ 5.269264] systemd[1]: Stopped target Socket Units. client # [ 5.270189] systemd[1]: Stopped target System Initialization. client # [ 5.271178] systemd[1]: Stopped target Swaps. client # [ 5.271966] systemd[1]: Stopped target Timer Units. worker1 # [ 5.252245] systemd[1]: Closed udev Control Socket. worker1 # [ 5.253026] systemd[1]: Starting Cleanup udev Database... lb # [ 5.221063] systemd[1]: initrd-cleanup.service: Deactivated successfully. lb # [ 5.222737] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker1 # [ 5.255181] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. client # [ 5.277703] systemd[1]: dbus.socket: Deactivated successfully. lb # [ 5.226060] systemd[1]: Stopped target Initrd Default Target. lb # [ 5.227574] systemd[1]: Stopped target Basic System. worker1 # [ 5.260320] systemd[1]: Stopped Create Static Device Nodes in /dev. client # [ 5.281042] systemd[1]: Closed D-Bus System Message Bus Socket. worker1 # [ 5.261214] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker1 # [ 5.262639] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. lb # [ 5.231097] systemd[1]: Stopped target Initrd Root Device. worker1 # [ 5.263616] systemd[1]: kmod-static-nodes.service: Deactivated successfully. lb # [ 5.232222] systemd[1]: Stopped target Path Units. client # [ 5.284964] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. lb # [ 5.233376] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. client # [ 5.287925] systemd[1]: Stopped Find NixOS closure. worker1 # [ 5.268221] systemd[1]: Stopped Create List of Static Device Nodes. lb # [ 5.239006] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. client # [ 5.294712] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 5.295699] systemd[1]: systemd-sysctl.service: Deactivated successfully. lb # [ 5.245191] systemd[1]: Stopped target Slice Units. lb # [ 5.245947] systemd[1]: Stopped target Socket Units. lb # [ 5.246700] systemd[1]: Stopped target System Initialization. client # [ 5.298833] systemd[1]: Stopped Apply Kernel Variables. worker2 # [ 5.348832] initrd-nixos-activation-start[176]: running activation script... lb # [ 5.247512] systemd[1]: Stopped target Swaps. lb # [ 5.252692] systemd[1]: Stopped target Timer Units. client # [ 5.304748] systemd[1]: systemd-modules-load.service: Deactivated successfully. lb # [ 5.254070] systemd[1]: dbus.socket: Deactivated successfully. lb # [ 5.254943] systemd[1]: Closed D-Bus System Message Bus Socket. lb # [ 5.255756] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. client # [ 5.309884] systemd[1]: Stopped Load Kernel Modules. lb # [ 5.259958] systemd[1]: Stopped Find NixOS closure. client # [ 5.312476] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. client # [ 5.316587] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker1 # [ 5.297063] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker1 # [ 5.298716] systemd[1]: Finished Cleanup udev Database. client # [ 5.318567] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker1 # [ 5.300488] systemd[1]: Reached target Switch Root. worker1 # [ 5.302458] systemd[1]: Starting NixOS Activation... lb # [ 5.271560] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 5.324059] systemd[1]: Stopped Create System Files and Directories. client # [ 5.325009] systemd[1]: Stopped target Local File Systems. lb # [ 5.273136] systemd[1]: systemd-sysctl.service: Deactivated successfully. lb # [ 5.274098] systemd[1]: Stopped Apply Kernel Variables. client # [ 5.325803] systemd[1]: Stopped target Preparation for Local File Systems. lb # [ 5.274863] systemd[1]: systemd-modules-load.service: Deactivated successfully. client # [ 5.326759] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. lb # [ 5.275848] systemd[1]: Stopped Load Kernel Modules. client # [ 5.327736] systemd[1]: Stopped Coldplug All udev Devices. client # [ 5.329854] systemd[1]: Stopping Rule-based Manager for Device Events and Files... client # [ 5.330920] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 5.331936] systemd[1]: Stopped Virtual Console Setup. lb # [ 5.279737] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. client # [ 5.332854] systemd[1]: initrd-cleanup.service: Deactivated successfully. client # [ 5.333802] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. client # [ 5.334724] systemd[1]: systemd-udevd.service: Deactivated successfully. client # [ 5.335636] systemd[1]: Stopped Rule-based Manager for Device Events and Files. lb # [ 5.288399] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. client # [ 5.342502] systemd[1]: systemd-udevd.service: Consumed 1.466s CPU time over 3.091s wall clock time, 22.3M memory peak. lb # [ 5.292354] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. lb # [ 5.293459] systemd[1]: Stopped Create System Files and Directories. lb # [ 5.294512] systemd[1]: Stopped target Local File Systems. lb # [ 5.296371] systemd[1]: Stopped target Preparation for Local File Systems. client # [ 5.348444] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. client # [ 5.349727] systemd[1]: Closed udev Control Socket. client # [ 5.350640] systemd[1]: Starting Cleanup udev Database... client # [ 5.351432] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. lb # [ 5.300171] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. lb # [ 5.301210] systemd[1]: Stopped Coldplug All udev Devices. lb # [ 5.301969] systemd[1]: Stopping Rule-based Manager for Device Events and Files... lb # [ 5.302983] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 5.303968] systemd[1]: Stopped Virtual Console Setup. client # [ 5.356229] systemd[1]: Stopped Create Static Device Nodes in /dev. client # [ 5.357102] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. client # [ 5.358215] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. client # [ 5.359196] systemd[1]: kmod-static-nodes.service: Deactivated successfully. lb # [ 5.308422] systemd[1]: systemd-udevd.service: Deactivated successfully. lb # [ 5.312114] systemd[1]: Stopped Rule-based Manager for Device Events and Files. client # [ 5.364556] systemd[1]: Stopped Create List of Static Device Nodes. lb # [ 5.313151] systemd[1]: systemd-udevd.service: Consumed 1.450s CPU time over 2.983s wall clock time, 22.3M memory peak. client # [ 5.365462] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 5.366483] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 5.314591] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. lb # [ 5.320396] systemd[1]: Closed udev Control Socket. lb # [ 5.321186] systemd[1]: Starting Cleanup udev Database... lb # [ 5.321962] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. lb # [ 5.323015] systemd[1]: Stopped Create Static Device Nodes in /dev. lb # [ 5.328447] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. lb # [ 5.329628] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. lb # [ 5.330627] systemd[1]: kmod-static-nodes.service: Deactivated successfully. lb # [ 5.331583] systemd[1]: Stopped Create List of Static Device Nodes. lb # [ 5.336569] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 5.337557] systemd[1]: Finished rw-sysroot-nix-store.service. client # [ 5.389390] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. client # [ 5.392755] systemd[1]: Finished Cleanup udev Database. client # [ 5.393969] systemd[1]: Reached target Switch Root. client # [ 5.395971] systemd[1]: Starting NixOS Activation... lb # [ 5.347888] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. lb # [ 5.349816] systemd[1]: Finished Cleanup udev Database. lb # [ 5.352357] systemd[1]: Reached target Switch Root. worker1 # [ 5.386101] initrd-nixos-activation-start[176]: booting system configuration /nix/store/6c2avm42395n27rpri261y0pclhhiysh-nixos-system-worker1-test lb # [ 5.356146] systemd[1]: Starting NixOS Activation... worker1 # [ 5.420461] initrd-nixos-activation-start[176]: running activation script... client # [ 5.484544] initrd-nixos-activation-start[179]: booting system configuration /nix/store/3zbq1wdsr2hpk2a2bj4l4l0k357dbi4h-nixos-system-client-test lb # [ 5.435893] initrd-nixos-activation-start[175]: booting system configuration /nix/store/kgjzg78npi7z07am2szqxkr41f06jp5w-nixos-system-lb-test client # [ 5.520110] initrd-nixos-activation-start[179]: running activation script... lb # [ 5.469442] initrd-nixos-activation-start[175]: running activation script... worker2 # [ 5.611149] initrd-nixos-activation-start[199]: setting up /etc... worker2 # [ 5.739855] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker2 # [ 5.743312] systemd[1]: Finished NixOS Activation. worker2 # [ 5.744749] systemd[1]: Starting Switch Root... worker1 # [ 5.678607] initrd-nixos-activation-start[199]: setting up /etc... worker2 # [ 5.769100] systemd[1]: Switching root. client # [ 5.765271] initrd-nixos-activation-start[202]: setting up /etc... lb # [ 5.719356] initrd-nixos-activation-start[198]: setting up /etc... worker1 # [ 5.803908] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker1 # [ 5.805287] systemd[1]: Finished NixOS Activation. worker1 # [ 5.806385] systemd[1]: Starting Switch Root... worker1 # [ 5.827549] systemd[1]: Switching root. client # [ 5.893297] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker2 # [ 5.959479] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). client # [ 5.896239] systemd[1]: Finished NixOS Activation. client # [ 5.897779] systemd[1]: Starting Switch Root... lb # [ 5.846401] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. lb # [ 5.849423] systemd[1]: Finished NixOS Activation. lb # [ 5.850663] systemd[1]: Starting Switch Root... client # [ 5.922236] systemd[1]: Switching root. lb # [ 5.871713] systemd[1]: Switching root. worker1 # [ 6.012419] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). client # [ 6.102693] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). lb # [ 6.058328] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker2 # [ 6.490984] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 6.503689] systemd[1]: Detected virtualization qemu. worker2 # [ 6.507084] systemd[1]: Detected architecture arm64. worker2 # [ 6.511307] systemd[1]: Detected first boot. worker2 # [ 6.517465] systemd[1]: Initializing machine ID from random generator. worker1 # [ 6.537493] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker1 # [ 6.543845] systemd[1]: Detected virtualization qemu. worker1 # [ 6.545867] systemd[1]: Detected architecture arm64. worker1 # [ 6.548741] systemd[1]: Detected first boot. worker1 # [ 6.554405] systemd[1]: Initializing machine ID from random generator. client # [ 6.631256] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 6.643888] systemd[1]: Detected virtualization qemu. client # [ 6.648092] systemd[1]: Detected architecture arm64. client # [ 6.650261] systemd[1]: Detected first boot. lb # [ 6.590583] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 6.655075] systemd[1]: Initializing machine ID from random generator. lb # [ 6.603617] systemd[1]: Detected virtualization qemu. lb # [ 6.606978] systemd[1]: Detected architecture arm64. lb # [ 6.611237] systemd[1]: Detected first boot. lb # [ 6.617394] systemd[1]: Initializing machine ID from random generator. worker2 # [ 6.852152] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 6.881672] systemd[1]: bpf-restrict-fs: LSM BPF program attached client # [ 6.986369] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 6.944848] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 7.085498] systemd[1]: Applying preset policy. worker1 # [ 7.092682] systemd[1]: Applying preset policy. client # [ 7.200756] systemd[1]: Applying preset policy. lb # [ 7.157734] systemd[1]: Applying preset policy. worker2 # [ 7.429542] systemd[1]: Populated /etc with preset unit settings. worker1 # [ 7.395310] systemd[1]: Populated /etc with preset unit settings. client # [ 7.472308] systemd[1]: Populated /etc with preset unit settings. lb # [ 7.452091] systemd[1]: Populated /etc with preset unit settings. worker2 # [ 7.657837] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker2 # [ 7.659123] systemd[1]: Stopped initrd-switch-root.service. worker2 # [ 7.663860] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker2 # [ 7.668374] systemd[1]: Created slice Slice /system/getty. worker2 # [ 7.670547] systemd[1]: Created slice User and Session Slice. worker2 # [ 7.671783] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker2 # [ 7.673870] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker2 # [ 7.676459] systemd[1]: Expecting device /dev/hvc0... worker2 # [ 7.678473] systemd[1]: Expecting device /dev/ttyAMA0... worker2 # [ 7.679501] systemd[1]: Reached target Local Encrypted Volumes. worker2 # [ 7.681114] systemd[1]: Stopped target initrd-fs.target. worker2 # [ 7.682578] systemd[1]: Stopped target initrd-root-fs.target. worker2 # [ 7.684975] systemd[1]: Stopped target initrd-switch-root.target. worker2 # [ 7.686152] systemd[1]: Reached target Virtual Machines and Containers. worker2 # [ 7.687847] systemd[1]: Reached target Path Units. worker2 # [ 7.689369] systemd[1]: Reached target Remote File Systems. worker2 # [ 7.691691] systemd[1]: Reached target Slice Units. worker2 # [ 7.692677] systemd[1]: Reached target Swaps. worker2 # [ 7.695780] systemd[1]: Listening on Query the User Interactively for a Password. worker1 # [ 7.625715] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker2 # [ 7.698845] systemd[1]: Listening on Process Core Dump Socket. worker1 # [ 7.627564] systemd[1]: Stopped initrd-switch-root.service. worker2 # [ 7.701172] systemd[1]: Listening on Credential Encryption/Decryption. worker2 # [ 7.703512] systemd[1]: Listening on Factory Reset Management. worker1 # [ 7.631927] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker2 # [ 7.704708] systemd[1]: Listening on Hostname Service Socket. worker1 # [ 7.634808] systemd[1]: Created slice Slice /system/getty. worker1 # [ 7.636953] systemd[1]: Created slice User and Session Slice. worker2 # [ 7.708933] systemd[1]: Starting Journal Log Access Socket... worker1 # [ 7.638212] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker2 # [ 7.711073] systemd[1]: Listening on Journal Audit Socket. worker2 # [ 7.713995] systemd[1]: Listening on Console Output Muting Service Socket. worker1 # [ 7.640330] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker1 # [ 7.640680] systemd[1]: Expecting device /dev/hvc0... worker2 # [ 7.715565] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker1 # [ 7.640946] systemd[1]: Expecting device /dev/ttyAMA0... worker1 # [ 7.641210] systemd[1]: Reached target Local Encrypted Volumes. worker1 # [ 7.641460] systemd[1]: Stopped target initrd-fs.target. worker2 # [ 7.717595] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.641696] systemd[1]: Stopped target initrd-root-fs.target. worker1 # [ 7.641926] systemd[1]: Stopped target initrd-switch-root.target. worker1 # [ 7.642174] systemd[1]: Reached target Virtual Machines and Containers. worker2 # [ 7.719327] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker1 # [ 7.642422] systemd[1]: Reached target Path Units. worker1 # [ 7.642651] systemd[1]: Reached target Remote File Systems. worker1 # [ 7.642874] systemd[1]: Reached target Slice Units. worker1 # [ 7.651100] systemd[1]: Reached target Swaps. worker2 # [ 7.725562] systemd[1]: Listening on Disk Repartitioning Service Socket. worker1 # [ 7.654991] systemd[1]: Listening on Query the User Interactively for a Password. worker2 # [ 7.726922] systemd[1]: Listening on udev Control Socket. worker2 # [ 7.728549] systemd[1]: Listening on udev Varlink Socket. worker1 # [ 7.658029] systemd[1]: Listening on Process Core Dump Socket. worker1 # [ 7.660338] systemd[1]: Listening on Credential Encryption/Decryption. worker2 # [ 7.732288] systemd[1]: Mounting Huge Pages File System... worker1 # [ 7.662732] systemd[1]: Listening on Factory Reset Management. worker1 # [ 7.664022] systemd[1]: Listening on Hostname Service Socket. worker2 # [ 7.737286] systemd[1]: Mounting POSIX Message Queue File System... worker1 # [ 7.668505] systemd[1]: Starting Journal Log Access Socket... worker1 # [ 7.670808] systemd[1]: Listening on Journal Audit Socket. worker1 # [ 7.674664] systemd[1]: Listening on Console Output Muting Service Socket. worker1 # [ 7.676284] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker2 # [ 7.748229] systemd[1]: Mounting Kernel Debug File System... client # [ 7.701271] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker1 # [ 7.678453] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os client # [ 7.702742] systemd[1]: Stopped initrd-switch-root.service. worker1 # [ 7.680547] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki client # [ 7.705930] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. client # [ 7.708988] systemd[1]: Created slice Slice /system/getty. worker1 # [ 7.686329] systemd[1]: Listening on Disk Repartitioning Service Socket. client # [ 7.711167] systemd[1]: Created slice User and Session Slice. worker1 # [ 7.688036] systemd[1]: Listening on udev Control Socket. worker2 # [ 7.761838] systemd[1]: Mounting Kernel Trace File System... worker1 # [ 7.689626] systemd[1]: Listening on udev Varlink Socket. client # [ 7.712447] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 7.714446] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker1 # [ 7.693382] systemd[1]: Mounting Huge Pages File System... client # [ 7.716047] systemd[1]: Expecting device /dev/hvc0... client # [ 7.717531] systemd[1]: Expecting device /dev/ttyAMA0... client # [ 7.719137] systemd[1]: Reached target Local Encrypted Volumes. client # [ 7.720699] systemd[1]: Stopped target initrd-fs.target. client # [ 7.722269] systemd[1]: Stopped target initrd-root-fs.target. client # [ 7.723812] systemd[1]: Stopped target initrd-switch-root.target. worker2 # [ 7.773379] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 7.700917] systemd[1]: Mounting POSIX Message Queue File System... client # [ 7.725461] systemd[1]: Reached target Virtual Machines and Containers. worker2 # [ 7.774674] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 7.727247] systemd[1]: Reached target Path Units. client # [ 7.728680] systemd[1]: Reached target Remote File Systems. client # [ 7.730294] systemd[1]: Reached target Slice Units. client # [ 7.731756] systemd[1]: Reached target Swaps. worker1 # [ 7.709456] systemd[1]: Mounting Kernel Debug File System... client # [ 7.734845] systemd[1]: Listening on Query the User Interactively for a Password. client # [ 7.737888] systemd[1]: Listening on Process Core Dump Socket. worker2 # [ 7.788067] systemd[1]: Mounting Kernel Configuration File System... client # [ 7.740236] systemd[1]: Listening on Credential Encryption/Decryption. worker2 # [ 7.789490] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm client # [ 7.742689] systemd[1]: Listening on Factory Reset Management. client # [ 7.743906] systemd[1]: Listening on Hostname Service Socket. worker1 # [ 7.721867] systemd[1]: Mounting Kernel Trace File System... client # [ 7.748401] systemd[1]: Starting Journal Log Access Socket... lb # [ 7.695415] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker2 # [ 7.797447] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore client # [ 7.750779] systemd[1]: Listening on Journal Audit Socket. lb # [ 7.697346] systemd[1]: Stopped initrd-switch-root.service. worker2 # [ 7.797838] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse client # [ 7.754683] systemd[1]: Listening on Console Output Muting Service Socket. lb # [ 7.701844] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. client # [ 7.756312] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. lb # [ 7.706268] systemd[1]: Created slice Slice /system/getty. client # [ 7.757916] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.735853] systemd[1]: Starting Create List of Static Device Nodes... lb # [ 7.708435] systemd[1]: Created slice User and Session Slice. worker1 # [ 7.736323] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 7.761147] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki lb # [ 7.709973] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 7.712664] systemd[1]: Started Forward Password Requests to Wall Directory Watch. client # [ 7.768704] systemd[1]: Listening on Disk Repartitioning Service Socket. lb # [ 7.715066] systemd[1]: Expecting device /dev/hvc0... lb # [ 7.717059] systemd[1]: Expecting device /dev/ttyAMA0... client # [ 7.771133] systemd[1]: Listening on udev Control Socket. lb # [ 7.719003] systemd[1]: Reached target Local Encrypted Volumes. worker2 # [ 7.819521] systemd[1]: Mounting FUSE Control File System... client # [ 7.772763] systemd[1]: Listening on udev Varlink Socket. worker2 # [ 7.819940] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 7.721161] systemd[1]: Stopped target initrd-fs.target. client # [ 7.776892] systemd[1]: Mounting Huge Pages File System... lb # [ 7.723106] systemd[1]: Stopped target initrd-root-fs.target. worker1 # [ 7.753309] systemd[1]: Mounting Kernel Configuration File System... lb # [ 7.725188] systemd[1]: Stopped target initrd-switch-root.target. worker1 # [ 7.753719] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm client # [ 7.780848] systemd[1]: Mounting POSIX Message Queue File System... worker1 # [ 7.753997] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore lb # [ 7.727250] systemd[1]: Reached target Virtual Machines and Containers. worker1 # [ 7.754263] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 7.729431] systemd[1]: Reached target Path Units. lb # [ 7.731372] systemd[1]: Reached target Remote File Systems. lb # [ 7.733456] systemd[1]: Reached target Slice Units. lb # [ 7.735406] systemd[1]: Reached target Swaps. lb # [ 7.738990] systemd[1]: Listening on Query the User Interactively for a Password. lb # [ 7.742286] systemd[1]: Listening on Process Core Dump Socket. client # [ 7.794734] systemd[1]: Mounting Kernel Debug File System... lb # [ 7.745150] systemd[1]: Listening on Credential Encryption/Decryption. lb # [ 7.748039] systemd[1]: Listening on Factory Reset Management. lb # [ 7.749391] systemd[1]: Listening on Hostname Service Socket. worker1 # [ 7.781328] systemd[1]: Mounting FUSE Control File System... lb # [ 7.754112] systemd[1]: Starting Journal Log Access Socket... client # [ 7.806521] systemd[1]: Mounting Kernel Trace File System... worker1 # [ 7.781764] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 7.756631] systemd[1]: Listening on Journal Audit Socket. worker2 # [ 7.859812] systemd[1]: Starting Journal Service... lb # [ 7.759805] systemd[1]: Listening on Console Output Muting Service Socket. lb # [ 7.761643] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. lb # [ 7.764079] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os client # [ 7.818435] systemd[1]: Starting Create List of Static Device Nodes... lb # [ 7.766713] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki client # [ 7.820788] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 7.873414] systemd[1]: Starting Load Kernel Modules... lb # [ 7.772591] systemd[1]: Listening on Disk Repartitioning Service Socket. lb # [ 7.774092] systemd[1]: Listening on udev Control Socket. lb # [ 7.775553] systemd[1]: Listening on udev Varlink Socket. lb # [ 7.779334] systemd[1]: Mounting Huge Pages File System... worker2 # [ 7.882798] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker1 # [ 7.811440] systemd[1]: Starting Journal Service... lb # [ 7.784926] systemd[1]: Mounting POSIX Message Queue File System... client # [ 7.845569] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 7.898312] systemd[1]: Starting Remount Root and Kernel File Systems... worker2 # [ 7.900449] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 7.800841] systemd[1]: Mounting Kernel Debug File System... client # [ 7.851857] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm client # [ 7.852270] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore client # [ 7.852550] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker1 # [ 7.834155] systemd[1]: Starting Load Kernel Modules... worker2 # [ 7.912861] systemd[1]: Starting Coldplug All udev Devices... lb # [ 7.811675] systemd[1]: Mounting Kernel Trace File System... worker2 # [ 7.920193] systemd[1]: Listening on Journal Log Access Socket. worker1 # [ 7.852617] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker2 # [ 7.924623] systemd[1]: Mounted Huge Pages File System. client # [ 7.880814] systemd[1]: Mounting FUSE Control File System... lb # [ 7.826683] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 7.858327] systemd[1]: Starting Remount Root and Kernel File Systems... worker2 # [ 7.929441] systemd[1]: Mounted POSIX Message Queue File System. worker1 # [ 7.860057] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 7.929996] systemd[1]: Mounted Kernel Debug File System. worker2 # [ 7.930379] systemd[1]: Mounted Kernel Trace File System. worker2 # [ 7.930751] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 7.931116] systemd[1]: Mounted FUSE Control File System. lb # [ 7.833065] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 7.886042] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker1 # [ 7.870198] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 7.874832] systemd[1]: Listening on Journal Log Access Socket. worker1 # [ 7.880975] systemd[1]: Mounted Huge Pages File System. lb # [ 7.852625] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 7.883443] systemd[1]: Mounted POSIX Message Queue File System. worker1 # [ 7.885667] systemd[1]: Mounted Kernel Debug File System. lb # [ 7.855219] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm client # [ 7.909560] systemd[1]: Starting Journal Service... worker1 # [ 7.888440] systemd[1]: Mounted Kernel Trace File System. worker2 # [ 7.962189] systemd[1]: Finished Create List of Static Device Nodes. worker1 # [ 7.891209] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 7.895287] systemd[1]: Mounted FUSE Control File System. worker2 # [ 7.969926] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 7.871338] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore client # [ 7.925434] systemd[1]: Starting Load Kernel Modules... lb # [ 7.873701] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse client # [ 7.943092] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker1 # [ 7.921768] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 7.894641] systemd[1]: Mounting FUSE Control File System... worker1 # [ 7.928091] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 7.905884] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker2 # [ 8.020075] EXT4-fs (vda): re-mounted 203ff707-c759-4c3a-a3f9-d5fbb101f69b. client # [ 7.970035] systemd[1]: Starting Remount Root and Kernel File Systems... client # [ 7.970449] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 8.028861] systemd[1]: Finished Remount Root and Kernel File Systems. lb # [ 7.929715] systemd[1]: Starting Journal Service... worker2 # [ 8.031664] systemd[1]: Listening on Disk Image Download Service Socket. worker2 # [ 8.034278] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore client # [ 7.993616] systemd[1]: Starting Coldplug All udev Devices... lb # [ 7.944932] systemd[1]: Starting Load Kernel Modules... worker2 # [ 8.047294] systemd[1]: Starting Load/Save OS Random Seed... worker2 # [ 8.047655] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 8.051773] systemd[1]: Finished Load Kernel Modules. client # [ 8.002607] systemd-journald[272]: Collecting audit messages is enabled. client # [ 8.007458] systemd[1]: Listening on Journal Log Access Socket. worker2 # [ 8.056370] systemd-journald[269]: Collecting audit messages is enabled. client # [ 8.009245] systemd[1]: Mounted Huge Pages File System. client # [ 8.010813] systemd[1]: Mounted POSIX Message Queue File System. worker1 # [ 7.987472] systemd[1]: Finished Load Kernel Modules. worker1 # [ 7.992149] systemd[1]: Starting Firewall... client # [ 8.017551] systemd[1]: Mounted Kernel Debug File System. worker2 # [ 8.065596] systemd[1]: Starting Firewall... client # [ 8.002660] systemd[1]: Queued start job for default target Multi-User System. client # [ 8.022172] systemd[1]: Started Journal Service. lb # [ 7.970626] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... client # [ 8.008816] systemd[1]: systemd-journald.service: Deactivated successfully. client # [ 8.009974] systemd[1]: Mounted Kernel Trace File System. worker1 # [ 8.005447] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 8.077836] systemd[1]: Starting Apply Kernel Variables... lb # [ 7.981544] systemd[1]: Starting Remount Root and Kernel File Systems... worker1 # [ 8.012038] systemd-journald[269]: Collecting audit messages is enabled. lb # [ 7.987425] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 8.024661] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 7.990003] systemd-journald[268]: Collecting audit messages is enabled. worker2 # [ 8.099280] systemd[1]: Started Journal Service. worker2 # [ 8.088213] systemd[1]: Queued start job for default target Multi-User System. worker2 # [ 8.089406] systemd[1]: systemd-journald.service: Deactivated successfully. lb # [ 7.989351] systemd[1]: Queued start job for default target Multi-User System. lb # [ 7.990714] systemd[1]: systemd-journald.service: Deactivated successfully. worker1 # [ 8.039132] EXT4-fs (vda): re-mounted 6fdfa4c4-3465-40d7-a9c6-192ad8c81d98. client # [ 8.050428] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker1 # [ 8.046666] systemd[1]: Finished Remount Root and Kernel File Systems. worker2 # [ 8.105589] systemd-modules-load[270]: Module 'atkbd' is built in lb # [ 8.021013] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 8.049382] systemd[1]: Listening on Disk Image Download Service Socket. worker1 # [ 8.049744] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore client # [ 8.063463] systemd-modules-load[273]: Module 'atkbd' is built in worker2 # [ 8.114141] systemd-modules-load[270]: Module 'loop' is built in client # [ 8.069512] systemd-modules-load[273]: Module 'loop' is built in worker1 # [ 8.063398] systemd[1]: Starting Load/Save OS Random Seed... worker1 # [ 8.065997] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 8.036459] systemd[1]: Started Journal Service. lb # [ 8.022181] systemd[1]: Listening on Journal Log Access Socket. worker2 # [ 8.124651] systemd[1]: Starting Flush Journal to Persistent Storage... lb # [ 8.024881] systemd[1]: Mounted Huge Pages File System. lb # [ 8.025745] systemd[1]: Mounted POSIX Message Queue File System. lb # [ 8.026745] systemd[1]: Mounted Kernel Debug File System. client # [ 8.083248] systemd[1]: Finished Load Kernel Modules. worker1 # [ 8.077505] systemd[1]: Started Journal Service. lb # [ 8.036564] systemd[1]: Mounted Kernel Trace File System. lb # [ 8.037553] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 8.038458] systemd[1]: Mounted Kernel Configuration File System. lb # [ 8.039576] systemd-modules-load[270]: Module 'atkbd' is built in client # [ 8.114299] EXT4-fs (vda): re-mounted 17878813-a2c4-493a-b0e7-fd3fa6804a97. client # [ 8.100817] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 8.084571] systemd[1]: Queued start job for default target Multi-User System. client # [ 8.105440] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 8.085937] systemd[1]: systemd-journald.service: Deactivated successfully. lb # [ 8.054939] systemd-modules-load[270]: Module 'loop' is built in lb # [ 8.055886] systemd[1]: Finished Load Kernel Modules. client # [ 8.111637] systemd[1]: Finished Remount Root and Kernel File Systems. worker1 # [ 8.095792] systemd-modules-load[270]: Module 'atkbd' is built in client # [ 8.117650] systemd[1]: Mounted FUSE Control File System. client # [ 8.121433] systemd[1]: Listening on Disk Image Download Service Socket. worker1 # [ 8.105113] systemd-modules-load[270]: Module 'loop' is built in lb # [ 8.075185] systemd[1]: Starting Firewall... worker2 # [ 8.177707] systemd-oomd[271]: No swap; memory pressure usage will be degraded lb # [ 8.080358] systemd[1]: Starting Apply Kernel Variables... client # [ 8.136496] systemd-oomd[275]: No swap; memory pressure usage will be degraded worker1 # [ 8.120264] systemd[1]: Starting Flush Journal to Persistent Storage... worker2 # [ 8.191109] systemd[1]: Finished Load/Save OS Random Seed. lb # [ 8.091533] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 8.147075] systemd[1]: Starting Flush Journal to Persistent Storage... worker2 # [ 8.197243] systemd[1]: Reached target First Boot Complete. lb # [ 8.096459] systemd[1]: Mounted FUSE Control File System. client # [ 8.149194] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 8.204409] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. client # [ 8.156191] systemd[1]: Starting Load/Save OS Random Seed... client # [ 8.157097] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 8.158298] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker2 # [ 8.223773] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 8.228593] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 8.257935] systemd-journald[269]: Received client request to flush runtime journal. worker1 # [ 8.177512] systemd-oomd[271]: No swap; memory pressure usage will be degraded lb # [ 8.168869] EXT4-fs (vda): re-mounted 98fd74e6-f0d8-4597-b85e-8a4553334efc. worker1 # [ 8.199099] systemd[1]: Finished Load/Save OS Random Seed. lb # [ 8.169827] systemd-oomd[271]: No swap; memory pressure usage will be degraded worker1 # [ 8.205831] systemd[1]: Reached target First Boot Complete. worker1 # [ 8.206739] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. lb # [ 8.180179] systemd[1]: Finished Remount Root and Kernel File Systems. lb # [ 8.181210] systemd[1]: Listening on Disk Image Download Service Socket. client # [ 8.233584] systemd[1]: Finished Load/Save OS Random Seed. client # [ 8.234513] systemd[1]: Reached target First Boot Complete. worker1 # [ 8.231740] systemd-journald[269]: Received client request to flush runtime journal. lb # [ 8.195068] systemd[1]: Starting Flush Journal to Persistent Storage... lb # [ 8.197370] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore lb # [ 8.207185] systemd[1]: Starting Load/Save OS Random Seed... worker2 # [ 8.309921] systemd[1]: Finished Apply Kernel Variables. client # [ 8.281266] systemd-journald[272]: Received client request to flush runtime journal. lb # [ 8.212202] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 8.213452] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker2 # [ 8.325093] systemd[1]: Finished Flush Journal to Persistent Storage. worker1 # [ 8.280453] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 8.281417] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 8.286120] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 8.292665] systemd[1]: Finished Flush Journal to Persistent Storage. client # [ 8.321120] systemd[1]: Finished Flush Journal to Persistent Storage. lb # [ 8.291317] systemd-journald[268]: Received client request to flush runtime journal. client # [ 8.343096] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 8.349308] systemd[1]: Starting Create Static Device Nodes in /dev... client # [ 8.353493] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 8.406908] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 8.409973] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 8.420518] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 8.342423] systemd[1]: Finished Apply Kernel Variables. lb # [ 8.343370] systemd[1]: Finished Load/Save OS Random Seed. lb # [ 8.350352] systemd[1]: Reached target First Boot Complete. lb # [ 8.351186] systemd[1]: Finished Flush Journal to Persistent Storage. lb # [ 8.393823] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 8.395763] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 8.434892] systemd[1]: Finished Create Static Device Nodes in /dev. worker1 # [ 8.435934] systemd[1]: Reached target Preparation for Local File Systems. worker1 # [ 8.448388] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker2 # [ 8.538386] systemd-udevd[306]: Using default interface naming scheme 'v261'. worker1 # [ 8.562930] systemd-udevd[310]: Using default interface naming scheme 'v261'. worker2 # [ 8.646271] systemd[1]: Mounting /run/wrappers... worker1 # [ 8.613694] systemd[1]: Mounting /run/wrappers... worker2 # [ 8.686979] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 8.739830] systemd[1]: Mounted /run/wrappers. worker2 # [ 8.742381] systemd[1]: Reached target Local File Systems. worker2 # [ 8.750158] systemd[1]: Listening on Boot Loader Control Service Socket. worker2 # [ 8.760397] systemd[1]: Starting register-nix-paths.service... worker2 # [ 8.768444] systemd[1]: Starting Create SUID/SGID Wrappers... worker1 # [ 8.713050] systemd[1]: Mounted /run/wrappers. lb # [ 8.681293] systemd[1]: Finished Create Static Device Nodes in /dev. worker1 # [ 8.713857] systemd[1]: Reached target Local File Systems. worker1 # [ 8.714707] systemd[1]: Listening on Boot Loader Control Service Socket. lb # [ 8.683336] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 8.783722] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker2 # [ 8.787974] systemd[1]: Starting Save Transient machine-id to Disk... worker2 # [ 8.789915] systemd[1]: Starting Create System Files and Directories... lb # [ 8.694027] systemd[1]: Mounting /run/wrappers... worker1 # [ 8.726164] systemd[1]: Starting register-nix-paths.service... worker1 # [ 8.727148] systemd[1]: Starting Create SUID/SGID Wrappers... worker1 # [ 8.727956] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. lb # [ 8.696863] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker1 # [ 8.756378] systemd[1]: Starting Save Transient machine-id to Disk... worker1 # [ 8.761349] systemd[1]: Starting Create System Files and Directories... worker1 # [ 8.766318] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 8.829891] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 8.830998] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 8.781333] systemd[1]: Mounted /run/wrappers. lb # [ 8.782182] systemd[1]: Reached target Local File Systems. client # [ 8.835918] systemd[1]: Mounting /run/wrappers... client # [ 8.840842] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 8.794156] systemd[1]: Listening on Boot Loader Control Service Socket. lb # [ 8.800206] systemd[1]: Starting register-nix-paths.service... lb # [ 8.806437] systemd[1]: Starting Create SUID/SGID Wrappers... lb # [ 8.807369] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. lb # [ 8.816339] systemd[1]: Starting Save Transient machine-id to Disk... worker2 # [ 8.931427] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker2 # [ 8.938791] systemd[1]: Finished Save Transient machine-id to Disk. lb # [ 8.839939] systemd[1]: Starting Create System Files and Directories... worker1 # [ 8.879186] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker1 # [ 8.885176] systemd[1]: Finished Save Transient machine-id to Disk. lb # [ 8.881089] systemd-udevd[308]: Using default interface naming scheme 'v261'. worker2 # [ 8.986817] systemd[1]: Finished Create System Files and Directories. worker2 # [ 8.991085] systemd[1]: Starting Rebuild Journal Catalog... client # [ 8.945943] systemd[1]: Finished Coldplug All udev Devices. worker2 # [ 9.001038] systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [ 8.955638] systemd[1]: Mounted /run/wrappers. client # [ 8.956991] systemd[1]: Reached target Local File Systems. client # [ 8.960100] systemd[1]: Listening on Boot Loader Control Service Socket. client # [ 8.964238] systemd[1]: Starting register-nix-paths.service... client # [ 8.971679] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 8.973428] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. client # [ 8.980234] systemd[1]: Starting Save Transient machine-id to Disk... client # [ 8.999116] systemd[1]: Starting Create System Files and Directories... worker1 # [ 8.979229] systemd[1]: Finished Create System Files and Directories. client # [ 9.005863] systemd-udevd[302]: Using default interface naming scheme 'v261'. worker1 # [ 8.989293] systemd[1]: Starting Rebuild Journal Catalog... worker1 # [ 8.998027] systemd[1]: Starting Record System Boot/Shutdown in UTMP... lb # [ 8.978338] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. lb # [ 8.989939] systemd[1]: Finished Save Transient machine-id to Disk. worker2 # [ 9.122326] systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 9.075862] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. client # [ 9.078886] systemd[1]: Finished Save Transient machine-id to Disk. lb # [ 9.067654] systemd[1]: Finished Create System Files and Directories. lb # [ 9.076528] systemd[1]: Starting Rebuild Journal Catalog... worker1 # [ 9.108940] systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 9.143150] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 9.093734] systemd[1]: Starting Record System Boot/Shutdown in UTMP... worker2 # [ 9.200704] systemd[1]: Finished Rebuild Journal Catalog. worker2 # [ 9.203980] systemd[1]: Starting Update is Completed... client # [ 9.166961] systemd[1]: Finished Create System Files and Directories. lb # [ 9.120413] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 9.174344] systemd[1]: Starting Rebuild Journal Catalog... client # [ 9.182761] systemd[1]: Starting Record System Boot/Shutdown in UTMP... worker1 # [ 9.174914] systemd[1]: Finished Rebuild Journal Catalog. worker1 # [ 9.177064] systemd[1]: Starting Update is Completed... worker2 # [ 9.291704] systemd[1]: Finished Update is Completed. worker1 # [ 9.266533] systemd[1]: Finished Update is Completed. client # [ 9.290907] systemd[1]: Finished Record System Boot/Shutdown in UTMP. lb # [ 9.257481] systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 9.337896] systemd[1]: Finished Rebuild Journal Catalog. client # [ 9.340383] systemd[1]: Starting Update is Completed... lb # [ 9.298016] systemd[1]: Finished Rebuild Journal Catalog. lb # [ 9.311682] systemd[1]: Starting Update is Completed... client # [ 9.405694] systemd[1]: Finished Update is Completed. lb # [ 9.415411] systemd[1]: Finished Update is Completed. client # [ 9.616882] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 9.649632] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker2 # [ 9.700231] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 9.709431] systemd[1]: Finished Create SUID/SGID Wrappers. lb # [ 9.638432] systemd[1]: Finished Coldplug All udev Devices. worker1 # [ 9.779702] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker1 # [ 9.786450] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 9.848197] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. client # [ 9.853458] systemd[1]: Finished Create SUID/SGID Wrappers. lb # [ 9.803809] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 9.889104] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. lb # [ 9.854266] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker2 # [ 9.967109] systemd[1]: Finished Firewall. client # [ 9.953935] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker1 # [ 10.037175] systemd[1]: Finished Firewall. lb # [ 10.080369] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. lb # [ 10.088475] systemd[1]: Finished Create SUID/SGID Wrappers. worker1 # [ 10.121830] systemd[1]: Finished Coldplug All udev Devices. client # [ 10.141903] (udev-worker)[360]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 10.197513] systemd[1]: Finished Coldplug All udev Devices. lb # [ 10.097494] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. client # [ 10.153174] (udev-worker)[349]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 10.161051] (udev-worker)[349]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 10.254203] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 10.192776] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 10.163389] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 10.282991] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker1 # [ 10.219491] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 10.211344] systemd[1]: Finished register-nix-paths.service. lb # [ 10.215092] systemd[1]: Reached target System Initialization. lb # [ 10.215967] systemd[1]: Started Discard unused filesystem blocks once a week. lb # [ 10.221889] systemd[1]: Started niks3 garbage collection timer. lb # [ 10.222828] systemd[1]: Started Daily Cleanup of Temporary Directories. lb # [ 10.223751] systemd[1]: Reached target Timer Units. lb # [ 10.231305] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 10.233684] systemd[1]: Listening on niks3 server socket. lb # [ 10.234508] systemd[1]: Listening on Nix Daemon Socket. lb # [ 10.235260] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. lb # [ 10.244426] systemd[1]: Reached target Socket Units. lb # [ 10.245203] systemd[1]: Reached target Basic System. lb # [ 10.245924] systemd[1]: Started backdoor.service. lb # [ 10.246640] systemd[1]: Starting Import lastlog data into lastlog2 database... lb # [ 10.257300] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 10.269927] systemd[1]: Starting Post-Boot Actions... lb # [ 10.294004] systemd[1]: Started Reset console on configuration changes. lb # [ 10.325889] systemd[1]: Starting resolvconf update... worker2 # [ 10.455465] systemd[1]: Finished register-nix-paths.service. worker2 # [ 10.457358] systemd[1]: Reached target System Initialization. worker2 # [ 10.459252] systemd[1]: Started Discard unused filesystem blocks once a week. worker1 # [ 10.390605] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker2 # [ 10.464867] systemd[1]: Started Daily Cleanup of Temporary Directories. client # [ 10.418836] systemd[1]: Condition check resulted in Virtio network device being skipped. worker2 # [ 10.470343] systemd[1]: Reached target Timer Units. client # [ 10.423729] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker1 # [ 10.407064] systemd[1]: Finished register-nix-paths.service. lb # [ 10.375418] systemd[1]: Started rustfs.service. client # [ 10.428162] systemd[1]: Starting Create SUID/SGID Wrappers... worker2 # [ 10.479059] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 10.429024] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker1 # [ 10.411405] systemd[1]: Reached target System Initialization. client # [ 10.430281] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker1 # [ 10.413551] systemd[1]: Started Discard unused filesystem blocks once a week. worker1 # [ 10.415713] systemd[1]: Started Daily Cleanup of Temporary Directories. worker1 # [ 10.421266] systemd[1]: Reached target Timer Units. worker2 # [ 10.491309] systemd[1]: Listening on Nix Daemon Socket. client # [ 10.442701] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker1 # [ 10.424795] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 10.445885] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 10.447176] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 10.400175] systemd[1]: Starting rustfs-bucket.service... worker2 # [ 10.504318] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 10.435436] systemd[1]: Listening on Nix Daemon Socket. worker2 # [ 10.510053] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker1 # [ 10.441771] systemd[1]: Listening on Nix worker-protocol over gRPC. client # [ 10.468986] systemd[1]: Finished register-nix-paths.service. worker2 # [ 10.519159] systemd[1]: Reached target Socket Units. client # [ 10.491069] mousedev: PS/2 mouse device common for all mice worker2 # [ 10.524115] systemd[1]: Reached target Basic System. worker2 # [ 10.529591] systemd[1]: Starting Import lastlog data into lastlog2 database... worker1 # [ 10.464257] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker2 # [ 10.539371] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 10.472196] systemd[1]: Reached target Socket Units. worker1 # [ 10.475853] systemd[1]: Reached target Basic System. worker1 # [ 10.484327] systemd[1]: Starting Import lastlog data into lastlog2 database... worker2 # [ 10.556774] systemd[1]: Starting Post-Boot Actions... worker2 # [ 10.564327] systemd[1]: Started Reset console on configuration changes. worker1 # [ 10.496679] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 10.470256] systemd[1]: Finished Post-Boot Actions. lb # connecting to host... worker2 # [ 10.572949] systemd[1]: Starting resolvconf update... worker1 # [ 10.505908] systemd[1]: Starting Post-Boot Actions... worker2 # [ 10.582680] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. lb # [ 10.485483] nsncd[450]: Sep 23 05:12:22.992 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 10.519895] systemd[1]: Started Reset console on configuration changes. worker2 # [ 10.594788] systemd[1]: Starting D-Bus System Message Bus... worker1 # [ 10.531025] systemd[1]: Starting resolvconf update... lb # [ 10.505290] systemd[1]: Started Name Service Cache Daemon (nsncd). worker1 # [ 10.539439] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 10.617195] systemd[1]: Finished Post-Boot Actions. worker1 # [ 10.552488] systemd[1]: Started backdoor.service. lb # [ 10.528678] (udev-worker)[388]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 10.565631] systemd[1]: Starting D-Bus System Message Bus... worker2 # [ 10.643384] systemd[1]: Started Name Service Cache Daemon (nsncd). lb # [ 10.544877] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 10.653637] systemd[1]: Reached target Host and Network Name Lookups. lb # [ 10.552518] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 10.589778] systemd[1]: Finished Post-Boot Actions. worker2 # [ 10.662584] nsncd[487]: Sep 23 05:12:23.097 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" lb # [ 10.580401] (udev-worker)[401]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 10.684464] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 10.691157] systemd[1]: Starting User Login Management... lb # [ 10.592640] systemd[1]: Starting D-Bus System Message Bus... lb # [ 10.593580] (udev-worker)[388]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 10.696415] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker1 # [ 10.631465] nsncd[488]: Sep 23 05:12:23.130 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 10.710682] systemd[1]: Started backdoor.service. worker1 # connecting to host... lb # [ 10.613741] systemd[1]: Starting User Login Management... worker1 # [ 10.650282] systemd[1]: Started Name Service Cache Daemon (nsncd). worker1 # [ 10.651265] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 10.727551] systemd[1]: Finished Import lastlog data into lastlog2 database. worker1 # [ 10.662557] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 10.668943] systemd[1]: Starting User Login Management... worker2 # [ 10.740766] dbus-broker-launch[491]: Looking up NSS user entry for 'systemd-timesync'... worker1 # [ 10.684576] systemd[1]: Finished Import lastlog data into lastlog2 database. worker2 # [ 10.758993] dbus-broker-launch[491]: NSS returned no entry for 'systemd-timesync' worker2 # [ 10.766053] dbus-broker-launch[491]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" lb # [ 10.676462] systemd[1]: Finished Import lastlog data into lastlog2 database. worker2 # [ 10.782025] systemd[1]: Started D-Bus System Message Bus. worker1 # [ 10.721349] dbus-broker-launch[493]: Looking up NSS user entry for 'systemd-timesync'... worker1 # [ 10.737783] dbus-broker-launch[493]: NSS returned no entry for 'systemd-timesync' worker1 # [ 10.738933] dbus-broker-launch[493]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker2 # connecting to host... worker1 # [ 10.760821] systemd[1]: Started D-Bus System Message Bus. worker2 # [ 10.841364] dbus-broker-launch[491]: Ready worker1 # [ 10.812609] dbus-broker-launch[493]: Ready worker2 # [ 10.907184] systemd-logind[508]: New seat seat0. worker2 # [ 10.911340] systemd[1]: Started User Login Management. worker1 # [ 10.847465] systemd-logind[505]: New seat seat0. worker2 # [ 10.917861] systemd[1]: Starting linger-users.service... worker1 # [ 10.855712] systemd[1]: Started User Login Management. worker1 # [ 10.859762] systemd[1]: Starting linger-users.service... worker2 # [ 10.936125] systemd[1]: Stopped target Host and Network Name Lookups. worker2 # [ 10.937110] systemd[1]: Stopping Host and Network Name Lookups... worker2 # [ 10.937960] systemd[1]: Stopped target User and Group Name Lookups. worker2 # [ 10.938811] systemd[1]: Stopping User and Group Name Lookups... worker2 # [ 10.939604] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker1 # [ 10.872434] systemd[1]: Stopped target Host and Network Name Lookups. worker1 # [ 10.873392] systemd[1]: Stopping Host and Network Name Lookups... worker1 # [ 10.874255] systemd[1]: Stopped target User and Group Name Lookups. worker1 # [ 10.875106] systemd[1]: Stopping User and Group Name Lookups... worker1 # [ 10.875907] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker1 # [ 10.896300] systemd[1]: nscd.service: Deactivated successfully. worker2 # [ 10.968337] systemd[1]: nscd.service: Deactivated successfully. worker2 # [ 10.972843] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker1 # [ 10.903987] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker1 # [ 10.905049] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 10.977017] systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 10.955889] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 11.006517] systemd[1]: linger-users.service: Deactivated successfully. client # [ 10.960424] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 10.961288] systemd[1]: Reached target System Initialization. client # [ 10.962137] systemd[1]: Started Discard unused filesystem blocks once a week. client # [ 10.963116] systemd[1]: Started Daily Cleanup of Temporary Directories. worker2 # [ 11.014003] systemd[1]: Finished linger-users.service. worker1 # [ 10.944346] (udev-worker)[347]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 10.972347] systemd[1]: Reached target Timer Units. client # [ 10.974904] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 10.978518] systemd[1]: Listening on Nix Daemon Socket. client # [ 10.981441] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker1 # [ 10.964224] systemd[1]: linger-users.service: Deactivated successfully. worker1 # [ 10.965280] systemd[1]: Finished linger-users.service. client # [ 10.988799] systemd[1]: Reached target Socket Units. lb # [ 10.936806] dbus-broker-launch[484]: Looking up NSS user entry for 'systemd-timesync'... client # [ 10.990866] systemd[1]: Reached target Basic System. client # [ 10.991761] systemd[1]: Started backdoor.service. worker1 # [ 10.972796] (udev-worker)[347]: Network interface NamePolicy= disabled on kernel command line. client # [ 10.993732] systemd[1]: Starting Import lastlog data into lastlog2 database... worker2 # [ 11.044469] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 11.045781] nsncd[561]: Sep 23 05:12:23.496 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 11.047735] systemd[1]: Reached target Host and Network Name Lookups. client # [ 11.000319] systemd[1]: Starting nftables firewall... lb # [ 10.953400] dbus-broker-launch[484]: NSS returned no entry for 'systemd-timesync' worker2 # [ 11.057634] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 11.058877] (udev-worker)[339]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 11.012835] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 10.993487] systemd[1]: Started Name Service Cache Daemon (nsncd). lb # [ 10.963173] dbus-broker-launch[484]: Invalid user-name in /nix/store/4yiprls1sll47jid3xzq0m9nmi1qz2ww-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker1 # [ 10.997369] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 11.068513] (udev-worker)[335]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 11.069714] (udev-worker)[339]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 11.000637] nsncd[560]: Sep 23 05:12:23.501 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 11.002628] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 11.003726] (udev-worker)[348]: Network interface NamePolicy= disabled on kernel command line. client # [ 11.030075] systemd[1]: Starting Post-Boot Actions... worker2 # [ 11.080385] systemd[1]: Finished resolvconf update. worker2 # [ 11.081167] systemd[1]: Reached target Preparation for Network. lb # [ 10.979806] systemd[1]: Stopped target Host and Network Name Lookups. worker1 # [ 11.018714] systemd[1]: Finished resolvconf update. worker2 # [ 11.089227] systemd[1]: Starting DHCP Client... worker1 # [ 11.019484] systemd[1]: Reached target Preparation for Network. lb # [ 10.987307] systemd[1]: Stopping Host and Network Name Lookups... lb # [ 10.990743] systemd[1]: Stopped target User and Group Name Lookups. worker2 # [ 11.092578] systemd[1]: Starting Extra networking commands.... lb # [ 10.991625] systemd[1]: Stopping User and Group Name Lookups... worker1 # [ 11.027050] systemd[1]: Starting DHCP Client... worker1 # [ 11.030652] systemd[1]: Starting Extra networking commands.... lb # [ 11.005272] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... lb # [ 11.006665] systemd-logind[489]: New seat seat0. lb # [ 11.007434] systemd[1]: Started User Login Management. client # [ 11.070629] systemd[1]: Started Reset console on configuration changes. lb # [ 11.019541] systemd[1]: nscd.service: Deactivated successfully. lb # [ 11.023970] systemd[1]: Stopped Name Service Cache Daemon (nsncd). client # [ 11.085943] systemd[1]: Starting resolvconf update... lb # [ 11.043534] systemd[1]: Starting linger-users.service... lb # [ 11.055983] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 11.060501] systemd[1]: Started D-Bus System Message Bus. client # [ 11.138435] systemd[1]: Starting D-Bus System Message Bus... client # connecting to host... lb # [ 11.107730] dbus-broker-launch[484]: Ready client # [ 11.186537] systemd[1]: Finished Post-Boot Actions. client: Guest shell says: b'Spawning backdoor root shell...\n' client # [ 11.215317] nsncd[455]: Sep 23 05:12:23.650 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 11.223068] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 11.255633] systemd[1]: Reached target Host and Network Name Lookups. client # [ 11.260939] systemd[1]: Reached target User and Group Name Lookups. lb # [ 11.211528] systemd[1]: linger-users.service: Deactivated successfully. lb # [ 11.218538] systemd[1]: Finished linger-users.service. lb # [ 11.224144] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 11.329924] dhcpcd[595]: dhcpcd-10.3.2 starting lb # [ 11.225329] nsncd[561]: Sep 23 05:12:23.730 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 11.287499] systemd[1]: Starting User Login Management... worker1 # [ 11.260607] dhcpcd[596]: dhcpcd-10.3.2 starting client: connected to guest root shell worker2 # [ 11.339756] dhcpcd[635]: dev: loaded udev lb # [ 11.236414] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 11.271731] dhcpcd[638]: dev: loaded udev client: (connecting took 11.70 seconds) client: (finished: waiting for the VM to finish booting, in 11.70 seconds) lb # [ 11.243058] systemd[1]: Reached target User and Group Name Lookups. lb # [ 11.247650] systemd-logind[489]: Watching system buttons on /dev/input/event0 (gpio-keys) lb # [ 11.254417] systemd[1]: Finished resolvconf update. lb # [ 11.265541] systemd[1]: Starting DHCP Client... worker2 # [ 11.392537] 8021q: 802.1Q VLAN Support v1.8 worker1 # [ 11.320862] 8021q: 802.1Q VLAN Support v1.8 client # [ 11.356513] systemd[1]: Finished Import lastlog data into lastlog2 database. worker1 # [ 11.348526] systemd[1]: Finished Extra networking commands.. worker1 # [ 11.349437] systemd[1]: Reached target Network. worker1 # [ 11.365383] systemd[1]: Starting Permit User Sessions... worker2 # [ 11.439250] systemd[1]: Finished Extra networking commands.. worker2 # [ 11.449823] systemd[1]: Reached target Network. worker2 # [ 11.452281] systemd[1]: Starting Permit User Sessions... worker1 # [ 11.392720] systemd-logind[505]: Watching system buttons on /dev/input/event0 (gpio-keys) worker2 # [ 11.466693] systemd[1]: Condition check resulted in Virtio network device being skipped. worker1 # [ 11.398403] systemd[1]: Condition check resulted in Virtio network device being skipped. worker1 # [ 11.420260] systemd[1]: Starting Address configuration of eth1... worker2 # [ 11.510481] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker2 # [ 11.496592] systemd[1]: Starting Address configuration of eth1... worker1 # [ 11.442442] cfg80211: Loading compiled-in X.509 certificates for regulatory database client # [ 11.454243] dbus-broker-launch[463]: Looking up NSS user entry for 'systemd-timesync'... lb # [ 11.418082] systemd[1]: Condition check resulted in Virtio network device being skipped. worker2 # [ 11.543661] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker2 # [ 11.544189] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker1 # [ 11.475238] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker1 # [ 11.475776] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker2 # [ 11.547297] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker2 # [ 11.547641] cfg80211: failed to load regulatory.db worker1 # [ 11.478837] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 client # [ 11.485811] dbus-broker-launch[463]: NSS returned no entry for 'systemd-timesync' worker1 # [ 11.479218] cfg80211: failed to load regulatory.db client # [ 11.486946] dbus-broker-launch[463]: Invalid user-name in /nix/store/6wvgzhp3xj1v1xybf1ixpbqi5gif5fbv-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker1 # [ 11.475765] systemd[1]: Finished Permit User Sessions. worker1 # [ 11.482163] systemd[1]: Started Getty on tty1. worker1 # [ 11.482963] systemd[1]: Reached target Login Prompts. worker2 # [ 11.569450] systemd[1]: Finished Permit User Sessions. worker2 # [ 11.574213] systemd[1]: Started Getty on tty1. worker2 # [ 11.574985] systemd[1]: Reached target Login Prompts. client # [ 11.537688] systemd-logind[485]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) worker2 # [ 11.593897] systemd-logind[508]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 11.548312] systemd-logind[485]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 11.549532] systemd-logind[485]: New seat seat0. client # [ 11.550483] systemd[1]: Started User Login Management. worker1 # [ 11.553769] 8021q: adding VLAN 0 to HW filter on device eth1 client # [ 11.571756] systemd[1]: Starting linger-users.service... worker2 # [ 11.640805] 8021q: adding VLAN 0 to HW filter on device eth1 client # [ 11.580203] systemd[1]: Started D-Bus System Message Bus. worker1 # [ 11.564729] network-addresses-eth1-start[651]: adding address 192.168.1.3/24... done worker1 # [ 11.583112] 8021q: adding VLAN 0 to HW filter on device eth0 worker1 # [ 11.570859] dhcpcd[638]: eth0: waiting for carrier worker2 # [ 11.659747] 8021q: adding VLAN 0 to HW filter on device eth0 worker1 # [ 11.575290] dhcpcd[638]: eth0: waiting for carrier worker2 # [ 11.646553] dhcpcd[635]: eth0: waiting for carrier worker1 # [ 11.578680] dhcpcd[638]: eth0: carrier acquired worker2 # [ 11.650225] dhcpcd[635]: eth0: waiting for carrier worker2 # [ 11.653000] network-addresses-eth1-start[651]: adding address 192.168.1.4/24... done worker1 # [ 11.584779] network-addresses-eth1-start[651]: adding address 2001:db8:1::3/64... done worker2 # [ 11.658557] dhcpcd[635]: eth0: carrier acquired worker1 # [ 11.596167] dhcpcd[638]: DUID 00:01:00:01:32:46:1c:38:52:54:00:12:34:56 worker1 # [ 11.597174] dhcpcd[638]: eth0: IAID 00:12:34:56 lb # [ 11.565606] dhcpcd[609]: dhcpcd-10.3.2 starting worker1 # [ 11.597806] dhcpcd[638]: eth0: adding address fe80::5054:ff:fe12:3456 worker2 # [ 11.669545] network-addresses-eth1-start[651]: adding address 2001:db8:1::4/64... done worker2 # [ 11.673493] dhcpcd[635]: DUID 00:01:00:01:32:46:1c:38:52:54:00:12:34:56 worker2 # [ 11.674449] dhcpcd[635]: eth0: IAID 00:12:34:56 worker2 # [ 11.675062] dhcpcd[635]: eth0: adding address fe80::5054:ff:fe12:3456 worker1 # [ 11.608138] systemd[1]: Finished Address configuration of eth1. lb # [ 11.577870] dhcpcd[624]: dev: loaded udev worker2 # [ 11.686832] systemd[1]: Finished Address configuration of eth1. client # [ 11.638952] systemd[1]: Stopped target Host and Network Name Lookups. client # [ 11.644111] systemd[1]: Stopping Host and Network Name Lookups... client # [ 11.645015] systemd[1]: Stopped target User and Group Name Lookups. client # [ 11.645877] systemd[1]: Stopping User and Group Name Lookups... client # [ 11.646700] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... client # [ 11.657732] dbus-broker-launch[463]: Ready client # [ 11.658462] systemd[1]: nscd.service: Deactivated successfully. client # [ 11.664495] systemd[1]: Stopped Name Service Cache Daemon (nsncd). lb # [ 11.651580] 8021q: 802.1Q VLAN Support v1.8 client # [ 11.688901] systemd[1]: linger-users.service: Deactivated successfully. client # [ 11.692280] systemd[1]: Finished linger-users.service. lb # [ 11.664904] mousedev: PS/2 mouse device common for all mice client # [ 11.707465] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 11.675904] systemd[1]: Finished Firewall. lb # [ 11.679750] systemd[1]: Reached target Preparation for Network. lb # [ 11.688153] systemd[1]: Starting Address configuration of eth1... worker1 # [ 11.734553] mousedev: PS/2 mouse device common for all mice lb # [ 11.700904] systemd[1]: Starting Extra networking commands.... worker1 # [ 11.752169] systemd-logind[505]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) client # [ 11.781018] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 11.785542] nsncd[550]: Sep 23 05:12:24.221 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 11.853457] mousedev: PS/2 mouse device common for all mice client # [ 11.792884] systemd[1]: Reached target Host and Network Name Lookups. client # [ 11.793828] systemd[1]: Reached target User and Group Name Lookups. client # [ 11.800174] systemd[1]: Finished resolvconf update. client # [ 11.804850] systemd[1]: Starting DHCP Client... worker2 # [ 11.885926] systemd-logind[508]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) lb # [ 11.863763] cfg80211: Loading compiled-in X.509 certificates for regulatory database lb # [ 11.874975] systemd-logind[489]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) lb # [ 11.930429] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' lb # [ 11.930939] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' lb # [ 11.937484] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 lb # [ 11.937819] cfg80211: failed to load regulatory.db lb # [ 11.954621] 8021q: adding VLAN 0 to HW filter on device eth1 client # [ 12.005005] dhcpcd[580]: dhcpcd-10.3.2 starting client # [ 12.014812] dhcpcd[587]: dev: loaded udev lb # [ 11.982944] network-addresses-eth1-start[633]: adding address 192.168.1.2/24... done client # [ 12.069806] 8021q: 802.1Q VLAN Support v1.8 lb # [ 12.015713] network-addresses-eth1-start[633]: adding address 2001:db8:1::2/64... done lb # [ 12.042416] dhcpcd[652]: /nix/store/1ki7bgdr0niilkg9jbkpcdaggbv0kb8a-openresolv-3.17.4/sbin/.resolvconf-wrapped: line 1250: kill: (638) - Operation not permitted lb # [ 12.050999] dhcpcd[652]: .resolvconf-wrapped: clearing stale lock pid 638 lb # [ 12.051994] systemd[1]: Finished Address configuration of eth1. client # [ 12.130662] cfg80211: Loading compiled-in X.509 certificates for regulatory database client # [ 12.186662] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker2 # [ 12.220265] dhcpcd[635]: eth0: soliciting a DHCP lease client # [ 12.187160] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' lb # [ 12.138451] 8021q: adding VLAN 0 to HW filter on device eth0 client # [ 12.190712] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 lb # [ 12.123179] dhcpcd[624]: eth0: waiting for carrier client # [ 12.191040] cfg80211: failed to load regulatory.db lb # [ 12.124777] dhcpcd[624]: eth0: carrier acquired worker2 # [ 12.228587] dhcpcd[635]: eth0: offered 10.0.2.15 from 10.0.2.2 worker2 # [ 12.236258] dhcpcd[635]: eth0: probing address 10.0.2.15/24 lb # [ 12.146626] dhcpcd[624]: DUID 00:01:00:01:32:46:1c:38:52:54:00:12:34:56 lb # [ 12.147635] dhcpcd[624]: eth0: IAID 00:12:34:56 lb # [ 12.153504] dhcpcd[624]: eth0: adding address fe80::5054:ff:fe12:3456 lb # [ 12.202877] systemd[1]: Finished Extra networking commands.. lb # [ 12.208232] systemd[1]: Reached target Network. lb # [ 12.213786] systemd[1]: Starting PostgreSQL Server... lb # [ 12.222785] systemd[1]: Starting Permit User Sessions... client # [ 12.294189] 8021q: adding VLAN 0 to HW filter on device eth0 client # [ 12.278679] dhcpcd[587]: eth0: waiting for carrier client # [ 12.279697] dhcpcd[587]: eth0: carrier acquired client # [ 12.291965] systemd[1]: Finished nftables firewall. client # [ 12.295995] systemd[1]: Reached target Preparation for Network. client # [ 12.302738] systemd[1]: Starting Address configuration of eth1... client # [ 12.306037] dhcpcd[587]: DUID 00:01:00:01:32:46:1c:38:52:54:00:12:34:56 client # [ 12.307062] dhcpcd[587]: eth0: IAID 00:12:34:56 client # [ 12.307696] dhcpcd[587]: eth0: adding address fe80::5054:ff:fe12:3456 client # [ 12.313258] systemd[1]: Starting Extra networking commands.... lb # [ 12.296750] systemd[1]: Finished Permit User Sessions. lb # [ 12.309399] systemd[1]: Started Getty on tty1. lb # [ 12.310145] systemd[1]: Reached target Login Prompts. worker1 # [ 12.383159] dhcpcd[638]: eth0: soliciting a DHCP lease client # [ 12.423584] 8021q: adding VLAN 0 to HW filter on device eth1 worker1 # [ 12.388524] dhcpcd[638]: eth0: offered 10.0.2.15 from 10.0.2.2 worker1 # [ 12.396225] dhcpcd[638]: eth0: probing address 10.0.2.15/24 client # [ 12.430473] network-addresses-eth1-start[623]: adding address 192.168.1.1/24... done worker2 # [ 12.519142] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 client # [ 12.456400] network-addresses-eth1-start[623]: adding address 2001:db8:1::1/64... done worker1 # [ 12.460278] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 client # [ 12.483651] systemd[1]: Finished Address configuration of eth1. lb # [ 12.499875] postgresql-pre-start[726]: The files belonging to this database system will be owned by user "postgres". lb # [ 12.503782] postgresql-pre-start[726]: This user must also own the server process. client # [ 12.560689] systemd[1]: Finished Extra networking commands.. client # [ 12.564775] systemd[1]: Reached target Network. client # [ 12.568745] systemd[1]: Starting Permit User Sessions... lb # [ 12.517242] postgresql-pre-start[726]: The database cluster will be initialized with locale "en_US.UTF-8". lb # [ 12.520907] postgresql-pre-start[726]: The default database encoding has accordingly been set to "UTF8". lb # [ 12.522169] postgresql-pre-start[726]: The default text search configuration will be set to "english". lb # [ 12.523394] postgresql-pre-start[726]: Data page checksums are enabled. lb # [ 12.528141] postgresql-pre-start[726]: fixing permissions on existing directory /var/lib/postgresql/18 ... ok lb # [ 12.529431] postgresql-pre-start[726]: creating subdirectories ... ok lb # [ 12.530274] postgresql-pre-start[726]: selecting dynamic shared memory implementation ... posix client # [ 12.628634] systemd[1]: Finished Permit User Sessions. client # [ 12.638012] systemd[1]: Started Getty on tty1. client # [ 12.638777] systemd[1]: Reached target Login Prompts. client # [ 12.687950] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 lb # [ 12.653410] postgresql-pre-start[726]: selecting default "max_connections" ... 100 client: (finished: waiting for unit network-addresses-eth1.service, in 13.15 seconds) client: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; } worker2 # [ 12.820629] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker2 # [ 12.831697] systemd[1]: Starting Virtual Console Setup... worker1 # [ 12.764305] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker1 # [ 12.767851] systemd[1]: Starting Virtual Console Setup... worker2 # [ 12.854147] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 12.786491] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 12.857550] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 12.859687] systemd[1]: Starting Virtual Console Setup... worker1 # [ 12.791023] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 12.795200] systemd[1]: Starting Virtual Console Setup... lb # [ 12.777281] postgresql-pre-start[726]: selecting default "shared_buffers" ... 128MB client: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; }, in 0.10 seconds) lb: waiting for unit network-addresses-eth1.service lb: waiting for the VM to finish booting lb: Guest shell says: b'Spawning backdoor root shell...\n' lb: connected to guest root shell lb: (connecting took 0.00 seconds) lb: (finished: waiting for the VM to finish booting, in 0.00 seconds) worker1 # [ 12.854389] systemd-logind[505]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker2 # [ 12.930594] systemd-logind[508]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) lb: (finished: waiting for unit network-addresses-eth1.service, in 0.08 seconds) lb: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; } worker1 # [ 12.931346] systemd-vconsole-setup[692]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 12.933996] systemd[1]: Finished Virtual Console Setup. worker2 # [ 13.010236] systemd-vconsole-setup[692]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 13.015017] systemd[1]: Finished Virtual Console Setup. lb: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; }, in 0.05 seconds) worker1: waiting for unit network-addresses-eth1.service worker1: waiting for the VM to finish booting worker1: Guest shell says: b'Spawning backdoor root shell...\n' worker1: connected to guest root shell worker1: (connecting took 0.00 seconds) worker1: (finished: waiting for the VM to finish booting, in 0.00 seconds) worker1: (finished: waiting for unit network-addresses-eth1.service, in 0.03 seconds) worker1: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; } worker1: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; }, in 0.02 seconds) worker2: waiting for unit network-addresses-eth1.service worker2: waiting for the VM to finish booting worker2: Guest shell says: b'Spawning backdoor root shell...\n' worker2: connected to guest root shell worker2: (connecting took 0.00 seconds) worker2: (finished: waiting for the VM to finish booting, in 0.00 seconds) worker2: (finished: waiting for unit network-addresses-eth1.service, in 0.03 seconds) worker2: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; } client # [ 13.057521] systemd[1]: Starting Virtual Console Setup... worker2: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; }, in 0.02 seconds) lb: waiting for unit niks3.service client # [ 13.082893] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 13.089218] systemd[1]: Stopped Virtual Console Setup. client # [ 13.093546] systemd[1]: Starting Virtual Console Setup... client # [ 13.122166] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. client # [ 13.178949] systemd-logind[485]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) client # [ 13.258113] systemd-vconsole-setup[700]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 13.262412] systemd[1]: Finished Virtual Console Setup. lb # [ 13.240088] dhcpcd[624]: eth0: soliciting a DHCP lease lb # [ 13.244498] dhcpcd[624]: eth0: offered 10.0.2.15 from 10.0.2.2 lb # [ 13.252199] dhcpcd[624]: eth0: probing address 10.0.2.15/24 lb # [ 13.323760] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 client # [ 13.619597] dhcpcd[587]: eth0: soliciting a DHCP lease client # [ 13.625121] dhcpcd[587]: eth0: offered 10.0.2.15 from 10.0.2.2 client # [ 13.628489] dhcpcd[587]: eth0: probing address 10.0.2.15/24 worker2 # [ 13.708674] dhcpcd[635]: eth0: soliciting an IPv6 router worker2 # [ 13.710530] dhcpcd[635]: eth0: Router Advertisement from fe80::2 worker2 # [ 13.711717] dhcpcd[635]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker2 # [ 13.713336] dhcpcd[635]: eth0: adding route to fec0::/64 worker2 # [ 13.714403] dhcpcd[635]: eth0: adding default route via fe80::2 lb # [ 13.627159] systemd[1]: Starting Virtual Console Setup... worker1 # [ 13.670378] dhcpcd[638]: eth0: soliciting an IPv6 router worker1 # [ 13.674566] dhcpcd[638]: eth0: Router Advertisement from fe80::2 worker1 # [ 13.677464] dhcpcd[638]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker1 # [ 13.680747] dhcpcd[638]: eth0: adding route to fec0::/64 worker1 # [ 13.683181] dhcpcd[638]: eth0: adding default route via fe80::2 lb # [ 13.656276] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. lb # [ 13.673836] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 13.679181] systemd[1]: Stopped Virtual Console Setup. lb # [ 13.692152] systemd[1]: Starting Virtual Console Setup... lb # [ 13.751046] systemd-logind[489]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) lb # [ 13.921314] systemd-vconsole-setup[757]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 13.925720] systemd[1]: Finished Virtual Console Setup. lb # [ 14.248276] dhcpcd[624]: eth0: soliciting an IPv6 router lb # [ 14.249139] dhcpcd[624]: eth0: Router Advertisement from fe80::2 lb # [ 14.249993] dhcpcd[624]: eth0: adding address fec0::5054:ff:fe12:3456/64 lb # [ 14.250900] dhcpcd[624]: eth0: adding route to fec0::/64 lb # [ 14.251618] dhcpcd[624]: eth0: adding default route via fe80::2 lb # [ 14.341293] postgresql-pre-start[726]: selecting default time zone ... UTC lb # [ 14.344124] postgresql-pre-start[726]: creating configuration files ... ok lb # [ 14.569669] postgresql-pre-start[726]: running bootstrap script ... ok client # [ 14.670088] dhcpcd[587]: eth0: soliciting an IPv6 router client # [ 14.672332] dhcpcd[587]: eth0: Router Advertisement from fe80::2 client # [ 14.673805] dhcpcd[587]: eth0: adding address fec0::5054:ff:fe12:3456/64 client # [ 14.675292] dhcpcd[587]: eth0: adding route to fec0::/64 client # [ 14.676734] dhcpcd[587]: eth0: adding default route via fe80::2 lb # [ 15.073673] postgresql-pre-start[726]: performing post-bootstrap initialization ... ok lb # [ 15.257899] postgresql-pre-start[726]: syncing data to disk ... ok lb # [ 15.259029] postgresql-pre-start[726]: initdb: warning: enabling "trust" authentication for local connections lb # [ 15.261583] postgresql-pre-start[726]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. lb # [ 15.263758] postgresql-pre-start[726]: Success. You can now start the database server using: lb # [ 15.264984] postgresql-pre-start[726]: pg_ctl -D /var/lib/postgresql/18 -l logfile start lb # [ 15.421083] postgres[779]: [779] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit lb # [ 15.424158] postgres[779]: [779] LOG: listening on IPv6 address "::1", port 5432 lb # [ 15.425622] postgres[779]: [779] LOG: listening on IPv4 address "127.0.0.1", port 5432 lb # [ 15.427474] postgres[779]: [779] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432" lb # [ 15.440422] postgres[800]: [800] LOG: database system was shut down at 2026-09-23 05:12:27 GMT lb # [ 15.446281] postgres[779]: [779] LOG: database system is ready to accept connections lb # [ 15.451121] systemd[1]: Started PostgreSQL Server. lb # [ 15.460322] systemd[1]: Starting PostgreSQL Setup Scripts... lb # [ 15.628114] postgresql-setup-start[811]: CREATE DATABASE lb # [ 15.669869] postgresql-setup-start[816]: CREATE ROLE lb # [ 15.688548] postgresql-setup-start[818]: ALTER DATABASE lb # [ 15.693491] systemd[1]: Finished PostgreSQL Setup Scripts. lb # [ 15.695652] systemd[1]: Reached target PostgreSQL. worker2 # [ 16.504278] dhcpcd[635]: eth0: leased 10.0.2.15 for 86400 seconds worker2 # [ 16.504610] dhcpcd[635]: eth0: adding route to 10.0.2.0/24 worker2 # [ 16.504749] dhcpcd[635]: eth0: adding default route via 10.0.2.2 worker2 # [ 16.677517] systemd[1]: Started DHCP Client. worker2 # [ 16.680565] systemd[1]: Reached target Multi-User System. worker2 # [ 16.682109] systemd[1]: Startup finished in 1.134s (kernel) + 4.914s (initrd) + 10.632s (userspace) = 16.681s. worker1 # [ 17.440428] dhcpcd[638]: eth0: leased 10.0.2.15 for 86400 seconds worker1 # [ 17.440744] dhcpcd[638]: eth0: adding route to 10.0.2.0/24 worker1 # [ 17.440880] dhcpcd[638]: eth0: adding default route via 10.0.2.2 worker1 # [ 17.583921] systemd[1]: Started DHCP Client. worker1 # [ 17.585596] systemd[1]: Reached target Multi-User System. worker1 # [ 17.587043] systemd[1]: Startup finished in 1.064s (kernel) + 5.037s (initrd) + 11.483s (userspace) = 17.586s. lb # [ 18.338944] dhcpcd[624]: eth0: leased 10.0.2.15 for 86400 seconds lb # [ 18.342252] dhcpcd[624]: eth0: adding route to 10.0.2.0/24 lb # [ 18.347073] dhcpcd[624]: eth0: adding default route via 10.0.2.2 client # [ 18.422906] dhcpcd[587]: eth0: leased 10.0.2.15 for 86400 seconds client # [ 18.426451] dhcpcd[587]: eth0: adding route to 10.0.2.0/24 client # [ 18.430968] dhcpcd[587]: eth0: adding default route via 10.0.2.2 lb # [ 18.519446] systemd[1]: Started DHCP Client. lb # [ 18.520759] systemd[1]: Reached target Network is Online. lb # [ 18.526467] systemd[1]: Started Envoy reverse proxy. lb # [ 18.528498] systemd[1]: Started mock-oidc.service. client # [ 18.582784] systemd[1]: Started DHCP Client. client # [ 18.585892] systemd[1]: Reached target Multi-User System. client # [ 18.587364] systemd[1]: Startup finished in 1.148s (kernel) + 5.045s (initrd) + 12.393s (userspace) = 18.586s. lb # [ 18.815333] mock-oidc-server[896]: Mock OIDC Server running lb # [ 18.817166] mock-oidc-server[896]: OIDC Address: 192.168.1.2:8080 lb # [ 18.818058] mock-oidc-server[896]: Issue Address: 0.0.0.0:8081 lb # [ 18.818874] mock-oidc-server[896]: Issuer: http://192.168.1.2:8080/oidc lb # [ 18.819752] mock-oidc-server[896]: JWKS: http://192.168.1.2:8080/oidc/.well-known/jwks.json lb # [ 18.822858] mock-oidc-server[896]: Discovery: http://192.168.1.2:8080/oidc/.well-known/openid-configuration lb # [ 18.825733] mock-oidc-server[896]: Issue tokens: http://0.0.0.0:8081/issue?sub=... lb # [ 18.955203] envoy[895]: [2026-09-23 05:12:31.465][895][info][main] [source/server/server.cc:467] initializing epoch 0 (base id=0, hot restart version=11.120) lb # [ 18.958498] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:469] statically linked extensions: lb # [ 18.961201] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.health_check.event_sinks: envoy.health_check.event_sink.file lb # [ 18.963036] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator lb # [ 18.966271] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest lb # [ 18.976157] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.filters.listener: envoy.filters.listener.dynamic_modules, envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.set_filter_state, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector lb # [ 19.003349] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.matching.action: envoy.matching.action.transport_socket.name, envoy.matching.actions.format_string, filter-chain-name lb # [ 19.007058] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 19.010350] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_mappers: envoy.tls.certificate_mappers.static_name, envoy.tls.upstream_certificate_mappers.filter_state_override lb # [ 19.013461] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.http.forward_client_cert_details: forward_client_cert lb # [ 19.015264] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor lb # [ 19.018618] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.filters.upstream_network: envoy.filters.upstream_network.reverse_tunnel_lifecycle lb # [ 19.021326] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.quic.connection_id_generator: envoy.quic.connection_id_generator.quic_lb, envoy.quic.deterministic_connection_id_generator lb # [ 19.023850] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.http.injected_credentials: envoy.http.injected_credentials.generic, envoy.http.injected_credentials.oauth2 lb # [ 19.026715] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.rbac.principals: envoy.rbac.principals.mtls_authenticated lb # [ 19.029142] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.ext_authz, envoy.filters.udp.session.http_capsule lb # [ 19.031892] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.geoip_providers: envoy.geoip_providers.maxmind lb # [ 19.033795] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.credential_injector, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec, envoy.filters.http.upstream_rbac, envoy.filters.http.wasm lb # [ 19.041323] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.generic_proxy.filters: envoy.filters.generic.router lb # [ 19.043168] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.upstream_options: envoy.extensions.upstreams.http.reverse_tunnel.v3.ReverseTunnelUpstreamCodecOptions, envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options lb # [ 19.047844] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector, envoy.upstream.local_address_selector.filter_state_override lb # [ 19.052965] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.matching.stat_matching_data.input: stat_full_name_match_input lb # [ 19.054903] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.bootstrap: envoy.bootstrap.dynamic_modules, envoy.bootstrap.internal_listener, envoy.bootstrap.reverse_tunnel.downstream_socket_interface, envoy.bootstrap.reverse_tunnel.upstream_socket_interface, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface, envoy.extensions.network.socket_interface.sockmap lb # [ 19.059943] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.matching.transport_socket.input: envoy.matching.inputs.endpoint_metadata, envoy.matching.inputs.locality_metadata, envoy.matching.inputs.transport_socket_filter_state lb # [ 19.063556] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso lb # [ 19.065830] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher lb # [ 19.067812] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.resource_monitors: envoy.resource_monitors.cgroup_memory, envoy.resource_monitors.cpu_utilization, envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource lb # [ 19.071277] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter lb # [ 19.073343] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.rds_factory: envoy.rds_factory.default lb # [ 19.074968] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter lb # [ 19.077067] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.regex_engines: envoy.regex_engines.google_re2 lb # [ 19.079030] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts lb # [ 19.082150] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme lb # [ 19.085435] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.matching.input_matchers: envoy.matching.input_matchers.generic_request_matcher, envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.dynamic_modules, envoy.matching.matchers.ip, envoy.matching.matchers.metadata_matcher, envoy.matching.matchers.runtime_fraction lb # [ 19.090265] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.health_checkers: envoy.health_checkers.dynamic_modules, envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift lb # [ 19.093709] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation lb # [ 19.095743] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.retry_priorities: envoy.retry_priorities.previous_priorities lb # [ 19.097959] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.access_loggers: envoy.access_loggers.dynamic_modules, envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stats, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log lb # [ 19.105258] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable lb # [ 19.107253] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.quic.client_packet_writer: envoy.quic.packet_writer.default lb # [ 19.109271] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.srds_factory: envoy.srds_factory.default lb # [ 19.110901] envoy[895]: [2026-09-23 05:12:31.467][895][info][main] [source/server/server.cc:471] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel, envoy.access_loggers.extension_filters.process_ratelimit lb # [ 19.113721] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.fault, envoy.filters.http.a2a, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.ai_protocol_manager, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.api_key_auth, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.bandwidth_share, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cache_v2, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_server, envoy.filters.http.file_system_buffer, envoy.filters.http.filter_chain, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_reverse_transcoder, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.mcp, envoy.filters.http.mcp_json_rest_bridge, envoy.filters.http.mcp_router, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.proto_api_scrubber, envoy.filters.http.proto_message_extraction, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.sse_to_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.thrift_to_metadata, envoy.filters.http.transform, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router lb # [ 19.145369] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.hashable_string, envoy.network.application_protocols, envoy.network.ip, envoy.network.network_namespace, envoy.network.transport_socket.http_11_proxy.info, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.ratelimit.hits_addend, envoy.router.debug_config, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port lb # [ 19.154107] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr lb # [ 19.156173] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy lb # [ 19.159075] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche lb # [ 19.162905] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.envelope, envoy.http.stateful_session.header lb # [ 19.165991] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.quic.server_preferred_address: quic.server_preferred_address.datasource, quic.server_preferred_address.fixed lb # [ 19.169584] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.http.cache_v2: envoy.extensions.http.cache_v2.file_system_http_cache, envoy.extensions.http.cache_v2.simple lb # [ 19.171928] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory lb # [ 19.175653] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.route_config_update_requester: envoy.route_config_update_requester.default lb # [ 19.177995] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] network.connection.client: default, envoy_internal lb # [ 19.179923] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.protocols: dubbo lb # [ 19.182163] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.upstreams: envoy.filters.connection_pools.tcp.generic lb # [ 19.186307] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.built_in_formatters: envoy.built_in_formatters.cel, envoy.built_in_formatters.http.default, envoy.built_in_formatters.metadata, envoy.built_in_formatters.stream_info.default, envoy.built_in_formatters.xfcc_value lb # [ 19.190289] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo, envoy.network.dns_resolver.hickory worker2 # [ 19.291885] systemd[1]: Reached target Network is Online. lb # [ 19.193169] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls lb # [ 19.200687] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.string_matcher: envoy.string_matcher.lua worker2 # [ 19.301967] systemd[1]: Starting scheduler-token.service... worker1 # [ 19.227997] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 19.204134] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 19.207029] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.dynamic_metadata, envoy.matching.inputs.dynamic_module_data_input, envoy.matching.inputs.filter_state, envoy.matching.inputs.local_reply, envoy.matching.inputs.network_namespace, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params lb # [ 19.217907] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.matching.stat_tag_matching_data.input: stat_tag_value_input lb # [ 19.219735] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.grpc_credentials: envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata lb # [ 19.222064] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.thrift_proxy.transports: auto, framed, header, unframed lb # [ 19.223835] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.tracers: envoy.tracers.datadog, envoy.tracers.dynamic_modules, envoy.tracers.fluentd, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin lb # [ 19.227294] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.quic.connection_debug_visitor: envoy.quic.connection_debug_visitor.basic, envoy.quic.connection_debug_visitor.quic_stats lb # [ 19.229888] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.clusters: envoy.cluster.dns, envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.composite, envoy.clusters.dynamic_forward_proxy, envoy.clusters.dynamic_modules, envoy.clusters.mcp_multicluster, envoy.clusters.redis, envoy.clusters.reverse_connection lb # [ 19.235127] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.matching.generic_proxy_request_input.input: envoy.matching.generic_proxy.input.host, envoy.matching.generic_proxy.input.method, envoy.matching.generic_proxy.input.path, envoy.matching.generic_proxy.input.property, envoy.matching.generic_proxy.input.request, envoy.matching.generic_proxy.input.service lb # [ 19.239742] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.load_balancing_policies: envoy.load_balancing_policies.client_side_weighted_round_robin, envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.dynamic_modules, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.load_aware_locality, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.override_host, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset, envoy.load_balancing_policies.wrr_locality lb # [ 19.247541] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.common.key_value: envoy.key_value.file_based lb # [ 19.249302] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.request_id: envoy.request_id.uuid lb # [ 19.250843] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.generic_proxy.codecs: envoy.generic_proxy.codecs.dubbo, envoy.generic_proxy.codecs.http1 lb # [ 19.253064] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.http.ext_proc.response_processors: envoy.extensions.http.ext_proc.save_processing_response lb # [ 19.255222] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.route.early_data_policy: envoy.route.early_data_policy.default lb # [ 19.257155] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8 lb # [ 19.259001] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain lb # [ 19.260845] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.dynamic_modules, envoy.tls.cert_validator.spiffe lb # [ 19.263425] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router lb # [ 19.266539] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router lb # [ 19.268261] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor lb # [ 19.270825] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.content_parsers: envoy.content_parsers.json lb # [ 19.272573] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.dynamic_modules, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd lb # [ 19.277569] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.dynamic_modules, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.ext_proc, envoy.filters.network.generic_proxy, envoy.filters.network.geoip, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.reverse_tunnel, envoy.filters.network.reverse_tunnel_drain_aware_http_connection_manager, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_bandwidth_limit, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy lb # [ 19.291336] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port lb # [ 19.293883] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.serializers: dubbo.hessian2 lb # [ 19.296832] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation lb # [ 19.299074] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple lb # [ 19.302691] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.http.header_validators: envoy.http.header_validators.envoy_default lb # [ 19.305041] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.ssl.server_context_factory: envoy.ssl.server_context_factory.default lb # [ 19.307089] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher worker2 # [ 19.415052] systemd[1]: scheduler-token.service: Deactivated successfully. lb # [ 19.309733] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.network_namespace, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san lb # [ 19.316855] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.tls.certificate_mappers: envoy.tls.certificate_mappers.sni, envoy.tls.certificate_mappers.static_name lb # [ 19.319087] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.connection_handler: envoy.connection_handler.default lb # [ 19.320858] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] quic.http_server_connection: quic.http_server_connection.default lb # [ 19.322627] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action worker2 # [ 19.424768] systemd[1]: Finished scheduler-token.service. lb # [ 19.324602] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.resolvers: envoy.ip, envoy.resolvers.reverse_connection lb # [ 19.326332] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.http.ext_proc.processing_request_modifiers: envoy.extensions.http.ext_proc.mapped_attribute_builder worker2 # [ 19.426755] systemd[1]: scheduler-token.service: Consumed 62ms CPU time over 116ms wall clock time, 3.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb # [ 19.328594] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment, envoy.tracers.opentelemetry.resource_detectors.static_config lb # [ 19.332089] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.tls.certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 19.334106] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.formatter: envoy.formatter.cel, envoy.formatter.dynamic_modules, envoy.formatter.file_content, envoy.formatter.generic_secret, envoy.formatter.metadata, envoy.formatter.req_without_query worker1 # [ 19.370104] nix-grpc-daemon[766]: ts=2026-09-23T05:12:31Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" lb # [ 19.337386] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua, envoy.router.cluster_specifier_plugin.matcher worker2 # [ 19.437095] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 19.339826] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.dynamic_modules, envoy.filters.udp_listener.udp_proxy lb # [ 19.342651] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff worker1 # [ 19.378885] nix-grpc-daemon[766]: ts=2026-09-23T05:12:31Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc lb # [ 19.345135] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.cel, envoy.tracers.opentelemetry.samplers.dynatrace, envoy.tracers.opentelemetry.samplers.parent_based, envoy.tracers.opentelemetry.samplers.trace_id_ratio_based worker1 # [ 19.381155] nix-grpc-daemon[766]: ts=2026-09-23T05:12:31Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 lb # [ 19.349331] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case lb # [ 19.351711] envoy[895]: [2026-09-23 05:12:31.468][895][info][main] [source/server/server.cc:471] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls lb # [ 19.356585] envoy[895]: [2026-09-23 05:12:31.486][895][info][main] [source/server/server.cc:531] HTTP header map info: worker1 # [ 19.392109] nix-grpc-daemon[766]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 19.395100] nix-grpc-daemon[766]: E0000 00:00:1790140351.898255 766 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 19.400610] nix-grpc-daemon[766]: E0000 00:00:1790140351.906959 766 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb # [ 19.357934] envoy[895]: [2026-09-23 05:12:31.486][895][info][main] [source/server/server.cc:534] request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-host,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-request-id lb # [ 19.373627] envoy[895]: [2026-09-23 05:12:31.486][895][info][main] [source/server/server.cc:534] request trailer map: 120 bytes: lb # [ 19.376265] envoy[895]: [2026-09-23 05:12:31.486][895][info][main] [source/server/server.cc:534] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id lb # [ 19.389783] envoy[895]: [2026-09-23 05:12:31.486][895][info][main] [source/server/server.cc:534] response trailer map: 144 bytes: grpc-message,grpc-status lb # [ 19.391895] envoy[895]: [2026-09-23 05:12:31.512][895][info][main] [source/server/server.cc:968] runtime: {} worker1 # [ 19.423078] nix-grpc-daemon[766]: ts=2026-09-23T05:12:31Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 19.425729] systemd[1]: Started Nix worker-protocol over gRPC. lb # [ 19.393315] envoy[895]: [2026-09-23 05:12:31.517][895][info][main] [source/server/server.cc:420] Raised file descriptor limits from 1024 to 524288. lb # [ 19.394987] envoy[895]: [2026-09-23 05:12:31.517][895][info][main] [source/server/server.cc:796] Starting admin HTTP server at 127.0.0.1:9901 lb # [ 19.396984] envoy[895]: [2026-09-23 05:12:31.517][895][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:9901 lb # [ 19.398867] envoy[895]: [2026-09-23 05:12:31.519][895][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration worker1 # [ 19.431932] nix-grpc-daemon[766]: E0000 00:00:1790140351.937133 808 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 19.401178] envoy[895]: [2026-09-23 05:12:31.519][895][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s) lb # [ 19.402898] envoy[895]: [2026-09-23 05:12:31.519][895][info][config] [source/server/configuration_impl.cc:138] loading 2 cluster(s) lb # [ 19.404674] envoy[895]: [2026-09-23 05:12:31.695][895][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s) lb # [ 19.406180] envoy[895]: [2026-09-23 05:12:31.713][895][info][config] [source/server/configuration_impl.cc:164] loading stats configuration lb # [ 19.407757] envoy[895]: [2026-09-23 05:12:31.714][895][warning][main] [source/server/server.cc:1038] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor. lb # [ 19.411327] envoy[895]: [2026-09-23 05:12:31.714][895][info][main] [source/server/server.cc:1077] starting main dispatch loop lb # [ 19.413078] envoy[895]: [2026-09-23 05:12:31.911][895][info][runtime] [source/common/runtime/runtime_impl.cc:567] RTDS has finished initialization lb # [ 19.414735] envoy[895]: [2026-09-23 05:12:31.911][895][info][upstream] [source/common/upstream/cluster_manager_impl.cc:250] cm init: all clusters initialized lb # [ 19.416628] envoy[895]: [2026-09-23 05:12:31.911][895][info][main] [source/server/server.cc:1057] all clusters initialized. initializing init manager lb # [ 19.420390] envoy[895]: [2026-09-23 05:12:31.929][895][info][config] [source/common/listener_manager/listener_manager_impl.cc:1067] all dependencies initialized. starting workers worker1 # [ 19.464142] nix-grpc-daemon[766]: E0000 00:00:1790140351.956644 808 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 19.470157] nix-grpc-daemon[766]: E0000 00:00:1790140351.957105 808 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 19.473250] nix-grpc-daemon[766]: E0000 00:00:1790140351.975710 808 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 19.481418] nix-grpc-daemon[766]: ts=2026-09-23T05:12:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 19.483556] nix-grpc-daemon[766]: ts=2026-09-23T05:12:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 19.585818] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 19.594455] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 19.596673] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 19.608517] nix-grpc-daemon[769]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 19.611442] nix-grpc-daemon[769]: E0000 00:00:1790140352.061251 769 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 19.616836] nix-grpc-daemon[769]: E0000 00:00:1790140352.069738 769 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 19.638076] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 19.641095] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 19.646221] nix-grpc-daemon[769]: E0000 00:00:1790140352.098117 812 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 19.664341] nix-grpc-daemon[769]: E0000 00:00:1790140352.117073 812 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 19.668062] nix-grpc-daemon[769]: E0000 00:00:1790140352.119988 812 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 19.686646] nix-grpc-daemon[769]: E0000 00:00:1790140352.138924 812 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 19.700307] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 19.702603] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 19.684758] nix-grpc-daemon[766]: ts=2026-09-23T05:12:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 19.689577] nix-grpc-daemon[766]: ts=2026-09-23T05:12:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 19.903830] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 19.909468] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 19.890893] nix-grpc-daemon[766]: ts=2026-09-23T05:12:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 19.896611] nix-grpc-daemon[766]: ts=2026-09-23T05:12:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.111014] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.116744] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.097553] nix-grpc-daemon[766]: ts=2026-09-23T05:12:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.103709] nix-grpc-daemon[766]: ts=2026-09-23T05:12:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.318576] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.325437] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.305090] nix-grpc-daemon[766]: ts=2026-09-23T05:12:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.311489] nix-grpc-daemon[766]: ts=2026-09-23T05:12:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.527119] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.533575] nix-grpc-daemon[769]: ts=2026-09-23T05:12:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.515023] nix-grpc-daemon[766]: ts=2026-09-23T05:12:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.519772] nix-grpc-daemon[766]: ts=2026-09-23T05:12:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.736523] nix-grpc-daemon[769]: ts=2026-09-23T05:12:33Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.740702] nix-grpc-daemon[769]: ts=2026-09-23T05:12:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.716203] nix-grpc-daemon[766]: ts=2026-09-23T05:12:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.721844] nix-grpc-daemon[766]: ts=2026-09-23T05:12:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.937716] nix-grpc-daemon[769]: ts=2026-09-23T05:12:33Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.943231] nix-grpc-daemon[769]: ts=2026-09-23T05:12:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.922932] nix-grpc-daemon[766]: ts=2026-09-23T05:12:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.928210] nix-grpc-daemon[766]: ts=2026-09-23T05:12:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.144698] nix-grpc-daemon[769]: ts=2026-09-23T05:12:33Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.150268] nix-grpc-daemon[769]: ts=2026-09-23T05:12:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.128962] nix-grpc-daemon[766]: ts=2026-09-23T05:12:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.133359] nix-grpc-daemon[766]: ts=2026-09-23T05:12:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.351680] nix-grpc-daemon[769]: ts=2026-09-23T05:12:33Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.357791] nix-grpc-daemon[769]: ts=2026-09-23T05:12:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.334445] nix-grpc-daemon[766]: ts=2026-09-23T05:12:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.340207] nix-grpc-daemon[766]: ts=2026-09-23T05:12:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.558980] nix-grpc-daemon[769]: ts=2026-09-23T05:12:34Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.563928] nix-grpc-daemon[769]: ts=2026-09-23T05:12:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.543351] nix-grpc-daemon[766]: ts=2026-09-23T05:12:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.548281] nix-grpc-daemon[766]: ts=2026-09-23T05:12:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.767001] nix-grpc-daemon[769]: ts=2026-09-23T05:12:34Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.771268] nix-grpc-daemon[769]: ts=2026-09-23T05:12:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.744298] nix-grpc-daemon[766]: ts=2026-09-23T05:12:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.749767] nix-grpc-daemon[766]: ts=2026-09-23T05:12:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.968875] nix-grpc-daemon[769]: ts=2026-09-23T05:12:34Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.974875] nix-grpc-daemon[769]: ts=2026-09-23T05:12:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.950955] nix-grpc-daemon[766]: ts=2026-09-23T05:12:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.957168] nix-grpc-daemon[766]: ts=2026-09-23T05:12:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.176326] nix-grpc-daemon[769]: ts=2026-09-23T05:12:34Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.181868] nix-grpc-daemon[769]: ts=2026-09-23T05:12:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.158043] nix-grpc-daemon[766]: ts=2026-09-23T05:12:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.163642] nix-grpc-daemon[766]: ts=2026-09-23T05:12:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.383129] nix-grpc-daemon[769]: ts=2026-09-23T05:12:34Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.389075] nix-grpc-daemon[769]: ts=2026-09-23T05:12:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.364668] nix-grpc-daemon[766]: ts=2026-09-23T05:12:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.370300] nix-grpc-daemon[766]: ts=2026-09-23T05:12:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.590194] nix-grpc-daemon[769]: ts=2026-09-23T05:12:35Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.595970] nix-grpc-daemon[769]: ts=2026-09-23T05:12:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.573135] nix-grpc-daemon[766]: ts=2026-09-23T05:12:35Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.577459] nix-grpc-daemon[766]: ts=2026-09-23T05:12:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.799581] nix-grpc-daemon[769]: ts=2026-09-23T05:12:35Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.804565] nix-grpc-daemon[769]: ts=2026-09-23T05:12:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.774489] nix-grpc-daemon[766]: ts=2026-09-23T05:12:35Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.780506] nix-grpc-daemon[766]: ts=2026-09-23T05:12:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.001356] nix-grpc-daemon[769]: ts=2026-09-23T05:12:35Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.007001] nix-grpc-daemon[769]: ts=2026-09-23T05:12:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.981440] nix-grpc-daemon[766]: ts=2026-09-23T05:12:35Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.987137] nix-grpc-daemon[766]: ts=2026-09-23T05:12:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.208736] nix-grpc-daemon[769]: ts=2026-09-23T05:12:35Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.214185] nix-grpc-daemon[769]: ts=2026-09-23T05:12:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.188422] nix-grpc-daemon[766]: ts=2026-09-23T05:12:35Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.194631] nix-grpc-daemon[766]: ts=2026-09-23T05:12:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.416091] nix-grpc-daemon[769]: ts=2026-09-23T05:12:35Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.422307] nix-grpc-daemon[769]: ts=2026-09-23T05:12:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.395859] nix-grpc-daemon[766]: ts=2026-09-23T05:12:35Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.402270] nix-grpc-daemon[766]: ts=2026-09-23T05:12:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.624435] nix-grpc-daemon[769]: ts=2026-09-23T05:12:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.629780] nix-grpc-daemon[769]: ts=2026-09-23T05:12:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.605026] nix-grpc-daemon[766]: ts=2026-09-23T05:12:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.609350] nix-grpc-daemon[766]: ts=2026-09-23T05:12:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 23.669525] rustfs-bucket-start[958]: mb s3://farm lb # [ 23.685525] systemd[1]: Finished rustfs-bucket.service. lb # [ 23.696786] systemd[1]: Starting niks3 server... worker2 # [ 23.833312] nix-grpc-daemon[769]: ts=2026-09-23T05:12:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.837646] nix-grpc-daemon[769]: ts=2026-09-23T05:12:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.805968] nix-grpc-daemon[766]: ts=2026-09-23T05:12:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.811726] nix-grpc-daemon[766]: ts=2026-09-23T05:12:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 23.848856] postgres[974]: [974] ERROR: relation "goose_db_version" does not exist at character 36 lb # [ 23.850189] postgres[974]: [974] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC lb # [ 23.876115] niks3-server[967]: 2026/09/23 05:12:36 OK 20241026095416_initial_model.sql (15.32ms) lb # [ 23.879787] niks3-server[967]: 2026/09/23 05:12:36 OK 20251210153512_drop_unused_gin_index.sql (4.03ms) lb # [ 23.885503] niks3-server[967]: 2026/09/23 05:12:36 OK 20251218171726_add_pins.sql (5.66ms) lb # [ 23.890300] niks3-server[967]: 2026/09/23 05:12:36 OK 20260628120000_add_object_size_and_stats.sql (4.75ms) lb # [ 23.895806] niks3-server[967]: 2026/09/23 05:12:36 OK 20260905000000_add_claims.sql (5.47ms) lb # [ 23.899881] niks3-server[967]: 2026/09/23 05:12:36 OK 20260920000000_drop_claims.sql (3.93ms) lb # [ 23.901930] niks3-server[967]: 2026/09/23 05:12:36 goose: successfully migrated database to version: 20260920000000 lb # [ 23.905626] niks3-server[967]: 2026/09/23 05:12:36 OK 1_commit_pending_closure.sql (5.73ms) lb # [ 23.908332] niks3-server[967]: 2026/09/23 05:12:36 OK 2_object_stats_trigger.sql (2.55ms) lb # [ 23.910349] niks3-server[967]: 2026/09/23 05:12:36 goose: up to current file version: 2 lb # [ 23.912112] niks3-server[967]: 2026/09/23 05:12:36 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key worker2 # [ 24.034477] nix-grpc-daemon[769]: ts=2026-09-23T05:12:36Z level=info event=scheduler_connected addr=lb:50051 lb # [ 23.934723] niks3-server[967]: 2026/09/23 05:12:36 INFO Read proxy enabled — serving cache objects from S3 worker2 # [ 24.040098] nix-grpc-daemon[769]: ts=2026-09-23T05:12:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 23.942476] niks3-server[967]: 2026/09/23 05:12:36 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 23.943937] niks3-server[967]: 2026/09/23 05:12:36 INFO systemd watchdog enabled interval=15s lb # [ 23.948389] niks3-server[967]: 2026/09/23 05:12:36 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 23.950003] systemd[1]: Started niks3 server. lb # [ 23.950922] systemd[1]: Reached target Multi-User System. lb # [ 23.951799] systemd[1]: Startup finished in 1.064s (kernel) + 5.085s (initrd) + 17.792s (userspace) = 23.941s. worker1 # [ 24.012578] nix-grpc-daemon[766]: ts=2026-09-23T05:12:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.018022] nix-grpc-daemon[766]: ts=2026-09-23T05:12:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.241256] nix-grpc-daemon[769]: ts=2026-09-23T05:12:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.246681] nix-grpc-daemon[769]: ts=2026-09-23T05:12:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.219015] nix-grpc-daemon[766]: ts=2026-09-23T05:12:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.224565] nix-grpc-daemon[766]: ts=2026-09-23T05:12:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.448122] nix-grpc-daemon[769]: ts=2026-09-23T05:12:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.453468] nix-grpc-daemon[769]: ts=2026-09-23T05:12:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: waiting for unit niks3.service, in 11.34 seconds) lb: waiting for TCP port 5751 on localhost worker1 # [ 24.425620] nix-grpc-daemon[766]: ts=2026-09-23T05:12:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.431015] nix-grpc-daemon[766]: ts=2026-09-23T05:12:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded! lb: (finished: waiting for TCP port 5751 on localhost, in 0.08 seconds) lb: waiting for unit envoy.service lb: (finished: waiting for unit envoy.service, in 0.05 seconds) lb: waiting for TCP port 50051 on localhost lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded! lb # [ 24.520124] envoy[895]: conn peer=[::1]:59276 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=4 lb: (finished: waiting for TCP port 50051 on localhost, in 0.03 seconds) worker2 # [ 24.654467] nix-grpc-daemon[769]: ts=2026-09-23T05:12:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.659098] nix-grpc-daemon[769]: ts=2026-09-23T05:12:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.636348] nix-grpc-daemon[766]: ts=2026-09-23T05:12:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.639100] nix-grpc-daemon[766]: ts=2026-09-23T05:12:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.793054] systemd[1]: Starting scheduler-token.service... worker2 # [ 24.860334] nix-grpc-daemon[769]: ts=2026-09-23T05:12:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.864324] nix-grpc-daemon[769]: ts=2026-09-23T05:12:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.891415] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 24.896349] systemd[1]: Finished scheduler-token.service. worker2 # [ 24.897508] systemd[1]: scheduler-token.service: Consumed 34ms CPU time over 103ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker1 # [ 24.837562] nix-grpc-daemon[766]: ts=2026-09-23T05:12:37Z level=info event=scheduler_connected addr=lb:50051 lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 24.848915] nix-grpc-daemon[766]: ts=2026-09-23T05:12:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.08 seconds) subtest: one node schedules and both builders hold a WorkerSession on it lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2 # [ 25.063953] nix-grpc-daemon[769]: ts=2026-09-23T05:12:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.069377] nix-grpc-daemon[769]: ts=2026-09-23T05:12:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.046355] nix-grpc-daemon[766]: ts=2026-09-23T05:12:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.051861] nix-grpc-daemon[766]: ts=2026-09-23T05:12:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.270723] nix-grpc-daemon[769]: ts=2026-09-23T05:12:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.276247] nix-grpc-daemon[769]: ts=2026-09-23T05:12:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.252858] nix-grpc-daemon[766]: ts=2026-09-23T05:12:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.258309] nix-grpc-daemon[766]: ts=2026-09-23T05:12:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.477629] nix-grpc-daemon[769]: ts=2026-09-23T05:12:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.483064] nix-grpc-daemon[769]: ts=2026-09-23T05:12:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.459010] nix-grpc-daemon[766]: ts=2026-09-23T05:12:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.464652] nix-grpc-daemon[766]: ts=2026-09-23T05:12:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.684176] nix-grpc-daemon[769]: ts=2026-09-23T05:12:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.689375] nix-grpc-daemon[769]: ts=2026-09-23T05:12:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.667590] nix-grpc-daemon[766]: ts=2026-09-23T05:12:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.671934] nix-grpc-daemon[766]: ts=2026-09-23T05:12:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.892654] nix-grpc-daemon[769]: ts=2026-09-23T05:12:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.896839] nix-grpc-daemon[769]: ts=2026-09-23T05:12:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.868309] nix-grpc-daemon[766]: ts=2026-09-23T05:12:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.873747] nix-grpc-daemon[766]: ts=2026-09-23T05:12:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 26.093878] nix-grpc-daemon[769]: ts=2026-09-23T05:12:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.099218] nix-grpc-daemon[769]: ts=2026-09-23T05:12:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 26.074819] nix-grpc-daemon[766]: ts=2026-09-23T05:12:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.080640] nix-grpc-daemon[766]: ts=2026-09-23T05:12:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.300782] nix-grpc-daemon[769]: ts=2026-09-23T05:12:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.306394] nix-grpc-daemon[769]: ts=2026-09-23T05:12:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.281623] nix-grpc-daemon[766]: ts=2026-09-23T05:12:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.287206] nix-grpc-daemon[766]: ts=2026-09-23T05:12:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.507718] nix-grpc-daemon[769]: ts=2026-09-23T05:12:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.513216] nix-grpc-daemon[769]: ts=2026-09-23T05:12:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.488254] nix-grpc-daemon[766]: ts=2026-09-23T05:12:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.493978] nix-grpc-daemon[766]: ts=2026-09-23T05:12:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.714246] nix-grpc-daemon[769]: ts=2026-09-23T05:12:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.719686] nix-grpc-daemon[769]: ts=2026-09-23T05:12:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.696940] nix-grpc-daemon[766]: ts=2026-09-23T05:12:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.701148] nix-grpc-daemon[766]: ts=2026-09-23T05:12:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.923007] nix-grpc-daemon[769]: ts=2026-09-23T05:12:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.927382] nix-grpc-daemon[769]: ts=2026-09-23T05:12:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.897791] nix-grpc-daemon[766]: ts=2026-09-23T05:12:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.903440] nix-grpc-daemon[766]: ts=2026-09-23T05:12:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 27.124677] nix-grpc-daemon[769]: ts=2026-09-23T05:12:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.130542] nix-grpc-daemon[769]: ts=2026-09-23T05:12:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.104453] nix-grpc-daemon[766]: ts=2026-09-23T05:12:39Z level=info event=scheduler_connected addr=lb:50051 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 27.109840] nix-grpc-daemon[766]: ts=2026-09-23T05:12:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.331676] nix-grpc-daemon[769]: ts=2026-09-23T05:12:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.337159] nix-grpc-daemon[769]: ts=2026-09-23T05:12:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.310942] nix-grpc-daemon[766]: ts=2026-09-23T05:12:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.316711] nix-grpc-daemon[766]: ts=2026-09-23T05:12:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.538466] nix-grpc-daemon[769]: ts=2026-09-23T05:12:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.544269] nix-grpc-daemon[769]: ts=2026-09-23T05:12:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.517553] nix-grpc-daemon[766]: ts=2026-09-23T05:12:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.523227] nix-grpc-daemon[766]: ts=2026-09-23T05:12:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.745628] nix-grpc-daemon[769]: ts=2026-09-23T05:12:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.751286] nix-grpc-daemon[769]: ts=2026-09-23T05:12:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.726595] nix-grpc-daemon[766]: ts=2026-09-23T05:12:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.730934] nix-grpc-daemon[766]: ts=2026-09-23T05:12:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.954688] nix-grpc-daemon[769]: ts=2026-09-23T05:12:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.959013] nix-grpc-daemon[769]: ts=2026-09-23T05:12:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.927125] nix-grpc-daemon[766]: ts=2026-09-23T05:12:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.932886] nix-grpc-daemon[766]: ts=2026-09-23T05:12:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.157004] nix-grpc-daemon[769]: ts=2026-09-23T05:12:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.165450] nix-grpc-daemon[769]: ts=2026-09-23T05:12:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 28.134022] nix-grpc-daemon[766]: ts=2026-09-23T05:12:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.148223] nix-grpc-daemon[766]: ts=2026-09-23T05:12:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2 # [ 28.366245] nix-grpc-daemon[769]: ts=2026-09-23T05:12:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.371604] nix-grpc-daemon[769]: ts=2026-09-23T05:12:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.342800] nix-grpc-daemon[766]: ts=2026-09-23T05:12:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.348549] nix-grpc-daemon[766]: ts=2026-09-23T05:12:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.573047] nix-grpc-daemon[769]: ts=2026-09-23T05:12:41Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.578382] nix-grpc-daemon[769]: ts=2026-09-23T05:12:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.549726] nix-grpc-daemon[766]: ts=2026-09-23T05:12:41Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.555428] nix-grpc-daemon[766]: ts=2026-09-23T05:12:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.779836] nix-grpc-daemon[769]: ts=2026-09-23T05:12:41Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.785354] nix-grpc-daemon[769]: ts=2026-09-23T05:12:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.758394] nix-grpc-daemon[766]: ts=2026-09-23T05:12:41Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.762689] nix-grpc-daemon[766]: ts=2026-09-23T05:12:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.988395] nix-grpc-daemon[769]: ts=2026-09-23T05:12:41Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.992670] nix-grpc-daemon[769]: ts=2026-09-23T05:12:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.959005] nix-grpc-daemon[766]: ts=2026-09-23T05:12:41Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.964484] nix-grpc-daemon[766]: ts=2026-09-23T05:12:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.189895] nix-grpc-daemon[769]: ts=2026-09-23T05:12:41Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.193967] nix-grpc-daemon[769]: ts=2026-09-23T05:12:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 29.165413] nix-grpc-daemon[766]: ts=2026-09-23T05:12:41Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.170743] nix-grpc-daemon[766]: ts=2026-09-23T05:12:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 29.395438] nix-grpc-daemon[769]: ts=2026-09-23T05:12:41Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.401043] nix-grpc-daemon[769]: ts=2026-09-23T05:12:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.371751] nix-grpc-daemon[766]: ts=2026-09-23T05:12:41Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.377490] nix-grpc-daemon[766]: ts=2026-09-23T05:12:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.602440] nix-grpc-daemon[769]: ts=2026-09-23T05:12:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.608072] nix-grpc-daemon[769]: ts=2026-09-23T05:12:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.578272] nix-grpc-daemon[766]: ts=2026-09-23T05:12:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.583689] nix-grpc-daemon[766]: ts=2026-09-23T05:12:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.809477] nix-grpc-daemon[769]: ts=2026-09-23T05:12:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.815127] nix-grpc-daemon[769]: ts=2026-09-23T05:12:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.786811] nix-grpc-daemon[766]: ts=2026-09-23T05:12:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.791063] nix-grpc-daemon[766]: ts=2026-09-23T05:12:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.018759] nix-grpc-daemon[769]: ts=2026-09-23T05:12:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.023149] nix-grpc-daemon[769]: ts=2026-09-23T05:12:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.987227] nix-grpc-daemon[766]: ts=2026-09-23T05:12:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.992630] nix-grpc-daemon[766]: ts=2026-09-23T05:12:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.219721] nix-grpc-daemon[769]: ts=2026-09-23T05:12:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.225229] nix-grpc-daemon[769]: ts=2026-09-23T05:12:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.193590] nix-grpc-daemon[766]: ts=2026-09-23T05:12:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.199235] nix-grpc-daemon[766]: ts=2026-09-23T05:12:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 30.426646] nix-grpc-daemon[769]: ts=2026-09-23T05:12:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.432253] nix-grpc-daemon[769]: ts=2026-09-23T05:12:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.400134] nix-grpc-daemon[766]: ts=2026-09-23T05:12:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.405674] nix-grpc-daemon[766]: ts=2026-09-23T05:12:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.633352] nix-grpc-daemon[769]: ts=2026-09-23T05:12:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.638896] nix-grpc-daemon[769]: ts=2026-09-23T05:12:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.606534] nix-grpc-daemon[766]: ts=2026-09-23T05:12:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.612110] nix-grpc-daemon[766]: ts=2026-09-23T05:12:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.840443] nix-grpc-daemon[769]: ts=2026-09-23T05:12:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.845941] nix-grpc-daemon[769]: ts=2026-09-23T05:12:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.814812] nix-grpc-daemon[766]: ts=2026-09-23T05:12:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.819367] nix-grpc-daemon[766]: ts=2026-09-23T05:12:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.049052] nix-grpc-daemon[769]: ts=2026-09-23T05:12:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.053283] nix-grpc-daemon[769]: ts=2026-09-23T05:12:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.015895] nix-grpc-daemon[766]: ts=2026-09-23T05:12:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.021433] nix-grpc-daemon[766]: ts=2026-09-23T05:12:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.250054] nix-grpc-daemon[769]: ts=2026-09-23T05:12:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.255541] nix-grpc-daemon[769]: ts=2026-09-23T05:12:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.222319] nix-grpc-daemon[766]: ts=2026-09-23T05:12:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.227650] nix-grpc-daemon[766]: ts=2026-09-23T05:12:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 31.459204] nix-grpc-daemon[769]: ts=2026-09-23T05:12:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.428919] nix-grpc-daemon[766]: ts=2026-09-23T05:12:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.434606] nix-grpc-daemon[766]: ts=2026-09-23T05:12:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.635586] nix-grpc-daemon[766]: ts=2026-09-23T05:12:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.641210] nix-grpc-daemon[766]: ts=2026-09-23T05:12:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.860676] nix-grpc-daemon[769]: ts=2026-09-23T05:12:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.866551] nix-grpc-daemon[769]: ts=2026-09-23T05:12:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.844310] nix-grpc-daemon[766]: ts=2026-09-23T05:12:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.848536] nix-grpc-daemon[766]: ts=2026-09-23T05:12:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.069904] nix-grpc-daemon[769]: ts=2026-09-23T05:12:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.074576] nix-grpc-daemon[769]: ts=2026-09-23T05:12:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.045038] nix-grpc-daemon[766]: ts=2026-09-23T05:12:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.050678] nix-grpc-daemon[766]: ts=2026-09-23T05:12:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.271545] nix-grpc-daemon[769]: ts=2026-09-23T05:12:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.277590] nix-grpc-daemon[769]: ts=2026-09-23T05:12:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.251810] nix-grpc-daemon[766]: ts=2026-09-23T05:12:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.257749] nix-grpc-daemon[766]: ts=2026-09-23T05:12:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 32.478956] nix-grpc-daemon[769]: ts=2026-09-23T05:12:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.484720] nix-grpc-daemon[769]: ts=2026-09-23T05:12:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 32.458725] nix-grpc-daemon[766]: ts=2026-09-23T05:12:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.465440] nix-grpc-daemon[766]: ts=2026-09-23T05:12:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.685591] nix-grpc-daemon[769]: ts=2026-09-23T05:12:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.691382] nix-grpc-daemon[769]: ts=2026-09-23T05:12:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.666727] nix-grpc-daemon[766]: ts=2026-09-23T05:12:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.672396] nix-grpc-daemon[766]: ts=2026-09-23T05:12:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.892157] nix-grpc-daemon[769]: ts=2026-09-23T05:12:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.897515] nix-grpc-daemon[769]: ts=2026-09-23T05:12:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.875458] nix-grpc-daemon[766]: ts=2026-09-23T05:12:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.879818] nix-grpc-daemon[766]: ts=2026-09-23T05:12:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.100780] nix-grpc-daemon[769]: ts=2026-09-23T05:12:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.105059] nix-grpc-daemon[769]: ts=2026-09-23T05:12:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.076437] nix-grpc-daemon[766]: ts=2026-09-23T05:12:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.081968] nix-grpc-daemon[766]: ts=2026-09-23T05:12:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.301842] nix-grpc-daemon[769]: ts=2026-09-23T05:12:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.307474] nix-grpc-daemon[769]: ts=2026-09-23T05:12:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.283002] nix-grpc-daemon[766]: ts=2026-09-23T05:12:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.288694] nix-grpc-daemon[766]: ts=2026-09-23T05:12:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.508719] nix-grpc-daemon[769]: ts=2026-09-23T05:12:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.514031] nix-grpc-daemon[769]: ts=2026-09-23T05:12:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 33.489686] nix-grpc-daemon[766]: ts=2026-09-23T05:12:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.495199] nix-grpc-daemon[766]: ts=2026-09-23T05:12:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 33.715146] nix-grpc-daemon[769]: ts=2026-09-23T05:12:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.720802] nix-grpc-daemon[769]: ts=2026-09-23T05:12:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.696161] nix-grpc-daemon[766]: ts=2026-09-23T05:12:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.701824] nix-grpc-daemon[766]: ts=2026-09-23T05:12:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.922030] nix-grpc-daemon[769]: ts=2026-09-23T05:12:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.927530] nix-grpc-daemon[769]: ts=2026-09-23T05:12:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.904666] nix-grpc-daemon[766]: ts=2026-09-23T05:12:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.908869] nix-grpc-daemon[766]: ts=2026-09-23T05:12:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.043753] nix-grpc-daemon[769]: ts=2026-09-23T05:12:46Z level=info event=scheduler_take_over lb # [ 33.942619] niks3-server[967]: 2026/09/23 05:12:46 INFO lead: acquired remote=192.168.1.4:38300 worker2 # [ 34.131461] nix-grpc-daemon[769]: ts=2026-09-23T05:12:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.136298] nix-grpc-daemon[769]: ts=2026-09-23T05:12:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.105769] nix-grpc-daemon[766]: ts=2026-09-23T05:12:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.111388] nix-grpc-daemon[766]: ts=2026-09-23T05:12:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.332467] nix-grpc-daemon[769]: ts=2026-09-23T05:12:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.338423] nix-grpc-daemon[769]: ts=2026-09-23T05:12:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.312463] nix-grpc-daemon[766]: ts=2026-09-23T05:12:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.317904] nix-grpc-daemon[766]: ts=2026-09-23T05:12:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.539592] nix-grpc-daemon[769]: ts=2026-09-23T05:12:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.545513] nix-grpc-daemon[769]: ts=2026-09-23T05:12:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.518850] nix-grpc-daemon[766]: ts=2026-09-23T05:12:47Z level=info event=scheduler_connected addr=lb:50051 lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 34.524584] nix-grpc-daemon[766]: ts=2026-09-23T05:12:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker2 # [ 34.747013] nix-grpc-daemon[769]: ts=2026-09-23T05:12:47Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.767310] nix-grpc-daemon[769]: ts=2026-09-23T05:12:47Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 34.772816] nix-grpc-daemon[769]: ts=2026-09-23T05:12:47Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:58724 worker2 # [ 34.779085] nix-grpc-daemon[769]: ts=2026-09-23T05:12:47Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 34.725777] nix-grpc-daemon[766]: ts=2026-09-23T05:12:47Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.797988] nix-grpc-daemon[769]: ts=2026-09-23T05:12:47Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:58724 worker2 # [ 34.803415] nix-grpc-daemon[769]: ts=2026-09-23T05:12:47Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open' worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.06 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 11.48 seconds) subtest: balancer auth: client cert, bearer token, nothing, unknown CN worker2 # [ 36.756835] nix-grpc-daemon[769]: ts=2026-09-23T05:12:49Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 36.770807] systemd[1]: Started Nix Daemon. worker2 # [ 36.937393] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 36.959891] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 77 ms (attempt 1/5) worker2 # [ 37.041671] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 192 ms (attempt 2/5) worker2 # [ 37.236353] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 266 ms (attempt 3/5) worker2 # [ 37.512722] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 279 ms (attempt 4/5) worker2 # [ 37.797934] nix-grpc-daemon[769]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 37.803205] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 37.769633] nix-grpc-daemon[766]: ts=2026-09-23T05:12:50Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 37.789367] systemd[1]: Started Nix Daemon. worker1 # [ 37.977925] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 38.000536] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 99 ms (attempt 1/5) worker1 # [ 38.101616] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 124 ms (attempt 2/5) worker1 # [ 38.227149] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 309 ms (attempt 3/5) worker1 # [ 38.543677] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 408 ms (attempt 4/5) worker1 # [ 38.958491] nix-grpc-daemon[766]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 38.964186] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 38.975943] nix-grpc-daemon[766]: ts=2026-09-23T05:12:51Z level=debug event=path_info_miss path=3zbq1wdsr2hpk2a2bj4l4l0k357dbi4h-nixos-system-client-test error="error: path '/nix/store/3zbq1wdsr2hpk2a2bj4l4l0k357dbi4h-nixos-system-client-test' is required, but there is no substituter that can build it" worker2 # [ 39.059954] nix-grpc-daemon[769]: ts=2026-09-23T05:12:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 39.077189] nix-grpc-daemon[769]: ts=2026-09-23T05:12:51Z level=debug event=path_info_miss path=3zbq1wdsr2hpk2a2bj4l4l0k357dbi4h-nixos-system-client-test error="error: path '/nix/store/3zbq1wdsr2hpk2a2bj4l4l0k357dbi4h-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.09 seconds) client: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key client: (finished: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds) lb # [ 39.187229] envoy[895]: conn peer=[2001:db8:1::1]:46308 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=407 ms=2364 worker2 # [ 39.446041] nix-grpc-daemon[769]: ts=2026-09-23T05:12:51Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:58726 worker1 # [ 39.411545] nix-grpc-daemon[766]: ts=2026-09-23T05:12:51Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 39.419025] nix-grpc-daemon[766]: ts=2026-09-23T05:12:51Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:49090 worker1 # [ 39.429249] nix-grpc-daemon[766]: ts=2026-09-23T05:12:51Z level=debug event=path_info_miss path=3zbq1wdsr2hpk2a2bj4l4l0k357dbi4h-nixos-system-client-test error="error: path '/nix/store/3zbq1wdsr2hpk2a2bj4l4l0k357dbi4h-nixos-system-client-test' is required, but there is no substituter that can build it" worker1 # [ 39.445448] nix-grpc-daemon[766]: ts=2026-09-23T05:12:51Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:49090 worker1 # [ 39.454210] nix-grpc-daemon[766]: ts=2026-09-23T05:12:51Z level=debug event=path_info_miss path=3zbq1wdsr2hpk2a2bj4l4l0k357dbi4h-nixos-system-client-test error="error: path '/nix/store/3zbq1wdsr2hpk2a2bj4l4l0k357dbi4h-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: rm -r /var/lib/nix-grpc-store client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.04 seconds) worker1 # [ 39.735614] nix-grpc-daemon[766]: ts=2026-09-23T05:12:52Z level=info event=denied method=QueryMissing cn=- role=none worker2 # [ 40.004815] nix-grpc-daemon[769]: ts=2026-09-23T05:12:52Z level=info event=denied method=QueryMissing cn=stranger role=none (finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.57 seconds) subtest: DAG build is scheduled across workers and lands in the cache client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 worker1 # [ 40.138806] nix-grpc-daemon[766]: ts=2026-09-23T05:12:52Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:49090 client # don't know how to build these paths: client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker1 # [ 40.153751] nix-grpc-daemon[766]: ts=2026-09-23T05:12:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 worker2 # [ 40.274764] nix-grpc-daemon[769]: ts=2026-09-23T05:12:52Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=0 worker2 # [ 40.282036] nix-grpc-daemon[769]: ts=2026-09-23T05:12:52Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 40.331757] nix-grpc-daemon[769]: ts=2026-09-23T05:12:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 40.353560] nix-grpc-daemon[769]: ts=2026-09-23T05:12:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58726 client # copying 1 paths to 'lb:50051'... worker2 # [ 40.373682] nix-grpc-daemon[769]: ts=2026-09-23T05:12:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'... worker2 # [ 40.398292] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 40.449736] nix-grpc-daemon[769]: ts=2026-09-23T05:12:52Z level=info event=niks3_push_started pid=870 worker2 # [ 40.655451] nix-grpc-daemon[870]: time=2026-09-23T05:12:53.108Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 40.788144] nix-daemon[855]: accepted connection from pid 883, user nix-grpc-daemon (trusted) worker2 # [ 40.811114] nix-daemon[855]: reaped child process 886, status = succeeded lb # [ 40.717850] niks3-server[967]: 2026/09/23 05:12:53 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 40.846632] nix-grpc-daemon[870]: time=2026-09-23T05:12:53.298Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 40.851337] nix-grpc-daemon[870]: time=2026-09-23T05:12:53.298Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)" lb # [ 40.779685] niks3-server[967]: 2026/09/23 05:12:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign lb # [ 40.836164] niks3-server[967]: 2026/09/23 05:12:53 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls lb # [ 40.844973] niks3-server[967]: 2026/09/23 05:12:53 INFO Signed narinfos id=1 count=1 worker2 # [ 40.951624] nix-grpc-daemon[870]: time=2026-09-23T05:12:53.402Z level=INFO msg="Uploading 1 narinfos" lb # [ 40.864667] niks3-server[967]: 2026/09/23 05:12:53 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst lb # [ 40.874517] niks3-server[967]: 2026/09/23 05:12:53 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete lb # [ 40.887136] niks3-server[967]: 2026/09/23 05:12:53 INFO Completed upload id=1 worker2 # [ 40.989451] nix-grpc-daemon[870]: time=2026-09-23T05:12:53.442Z level=INFO msg="Upload complete. (329ms)" lb # [ 40.896734] niks3-server[967]: 2026/09/23 05:12:53 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo worker2 # [ 41.000592] nix-grpc-daemon[769]: ts=2026-09-23T05:12:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=0 paths=1 nar_bytes_in=504 worker2 # [ 41.024825] nix-grpc-daemon[769]: ts=2026-09-23T05:12:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 41.035431] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 41.042939] nix-daemon[855]: reaped child process 868, status = succeeded client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051... client # farm-a> worker2: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker2 # [ 41.060272] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-t1 worker2 # [ 42.531861] nix-daemon[855]: reaped child process 890, status = succeeded worker2 # [ 42.668512] nix-daemon[855]: accepted connection from pid 907, user nix-grpc-daemon (trusted) worker2 # [ 42.690392] nix-daemon[855]: reaped child process 910, status = succeeded lb # [ 42.597037] niks3-server[967]: 2026/09/23 05:12:55 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 42.720595] nix-grpc-daemon[870]: time=2026-09-23T05:12:55.169Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 42.722586] nix-grpc-daemon[870]: time=2026-09-23T05:12:55.170Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)" lb # [ 42.652665] niks3-server[967]: 2026/09/23 05:12:55 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv lb # [ 42.664868] niks3-server[967]: 2026/09/23 05:12:55 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst lb # [ 42.676904] niks3-server[967]: 2026/09/23 05:12:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign worker2 # [ 42.784894] nix-grpc-daemon[870]: time=2026-09-23T05:12:55.236Z level=INFO msg="Uploading 1 narinfos" lb # [ 42.685867] niks3-server[967]: 2026/09/23 05:12:55 INFO Signed narinfos id=2 count=1 lb # [ 42.691868] niks3-server[967]: 2026/09/23 05:12:55 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls lb # [ 42.708441] niks3-server[967]: 2026/09/23 05:12:55 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete lb # [ 42.716422] niks3-server[967]: 2026/09/23 05:12:55 INFO Completed upload id=2 worker2 # [ 42.818704] nix-grpc-daemon[870]: time=2026-09-23T05:12:55.271Z level=INFO msg="Upload complete. (341ms)" lb # [ 42.720275] niks3-server[967]: 2026/09/23 05:12:55 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo worker2 # [ 42.845333] nix-grpc-daemon[769]: ts=2026-09-23T05:12:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=1 worker2 # [ 42.863841] nix-grpc-daemon[769]: ts=2026-09-23T05:12:55Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=0 worker2 # [ 42.868246] nix-grpc-daemon[769]: ts=2026-09-23T05:12:55Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 42.872302] nix-daemon[855]: reaped child process 888, status = succeeded worker2 # [ 42.887663] nix-grpc-daemon[769]: ts=2026-09-23T05:12:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 42.905486] nix-grpc-daemon[769]: ts=2026-09-23T05:12:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 42.960809] nix-grpc-daemon[870]: time=2026-09-23T05:12:55.413Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 42.973763] nix-grpc-daemon[769]: ts=2026-09-23T05:12:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'... worker2 # [ 42.997471] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 43.174485] nix-daemon[855]: accepted connection from pid 916, user nix-grpc-daemon (trusted) worker2 # [ 43.200128] nix-daemon[855]: reaped child process 919, status = succeeded lb # [ 43.101131] niks3-server[967]: 2026/09/23 05:12:55 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 43.214108] nix-grpc-daemon[870]: time=2026-09-23T05:12:55.666Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 43.215776] nix-grpc-daemon[870]: time=2026-09-23T05:12:55.666Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)" lb # [ 43.145501] niks3-server[967]: 2026/09/23 05:12:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign worker2 # [ 43.262081] nix-grpc-daemon[870]: time=2026-09-23T05:12:55.714Z level=INFO msg="Uploading 1 narinfos" lb # [ 43.159861] niks3-server[967]: 2026/09/23 05:12:55 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst lb # [ 43.170544] niks3-server[967]: 2026/09/23 05:12:55 INFO Signed narinfos id=3 count=1 lb # [ 43.175231] niks3-server[967]: 2026/09/23 05:12:55 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls lb # [ 43.190257] niks3-server[967]: 2026/09/23 05:12:55 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete lb # [ 43.204273] niks3-server[967]: 2026/09/23 05:12:55 INFO Completed upload id=3 worker2 # [ 43.306366] nix-grpc-daemon[870]: time=2026-09-23T05:12:55.759Z level=INFO msg="Upload complete. (288ms)" lb # [ 43.208645] niks3-server[967]: 2026/09/23 05:12:55 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo worker2 # [ 43.316496] nix-grpc-daemon[769]: ts=2026-09-23T05:12:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=0 paths=1 nar_bytes_in=624 worker2 # [ 43.332425] nix-grpc-daemon[769]: ts=2026-09-23T05:12:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 43.349223] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 43.360235] nix-daemon[855]: reaped child process 914, status = succeeded client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051... worker2 # [ 43.371283] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) client # farm-b> worker2: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-t1 worker2 # [ 44.869437] nix-daemon[855]: reaped child process 923, status = succeeded worker2 # [ 44.938668] nix-daemon[855]: accepted connection from pid 939, user nix-grpc-daemon (trusted) lb # [ 44.870465] niks3-server[967]: 2026/09/23 05:12:57 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 44.976437] nix-daemon[855]: reaped child process 942, status = succeeded worker2 # [ 44.986512] nix-grpc-daemon[870]: time=2026-09-23T05:12:57.438Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 44.988140] nix-grpc-daemon[870]: time=2026-09-23T05:12:57.438Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)" lb # [ 44.920719] niks3-server[967]: 2026/09/23 05:12:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign worker2 # [ 45.035843] nix-grpc-daemon[870]: time=2026-09-23T05:12:57.488Z level=INFO msg="Uploading 1 narinfos" lb # [ 44.941283] niks3-server[967]: 2026/09/23 05:12:57 INFO Signed narinfos id=4 count=1 lb # [ 44.949386] niks3-server[967]: 2026/09/23 05:12:57 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls lb # [ 44.959033] niks3-server[967]: 2026/09/23 05:12:57 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst lb # [ 44.979190] niks3-server[967]: 2026/09/23 05:12:57 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete lb # [ 44.988097] niks3-server[967]: 2026/09/23 05:12:57 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv lb # [ 44.993099] niks3-server[967]: 2026/09/23 05:12:57 INFO Completed upload id=4 worker2 # [ 45.094158] nix-grpc-daemon[870]: time=2026-09-23T05:12:57.547Z level=INFO msg="Upload complete. (307ms)" lb # [ 44.995526] niks3-server[967]: 2026/09/23 05:12:57 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo worker2 # [ 45.114000] nix-grpc-daemon[769]: ts=2026-09-23T05:12:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=1 worker2 # [ 45.132432] nix-grpc-daemon[769]: ts=2026-09-23T05:12:57Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=0 worker2 # [ 45.137182] nix-grpc-daemon[769]: ts=2026-09-23T05:12:57Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 45.141747] nix-daemon[855]: reaped child process 921, status = succeeded worker2 # [ 45.159020] nix-grpc-daemon[769]: ts=2026-09-23T05:12:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 45.176353] nix-grpc-daemon[769]: ts=2026-09-23T05:12:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 45.193108] nix-grpc-daemon[870]: time=2026-09-23T05:12:57.646Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 45.205674] nix-grpc-daemon[769]: ts=2026-09-23T05:12:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'... worker2 # [ 45.230527] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 45.395194] nix-daemon[855]: accepted connection from pid 949, user nix-grpc-daemon (trusted) worker2 # [ 45.421400] nix-daemon[855]: reaped child process 952, status = succeeded lb # [ 45.316833] niks3-server[967]: 2026/09/23 05:12:57 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 45.429038] nix-grpc-daemon[870]: time=2026-09-23T05:12:57.882Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 45.431496] nix-grpc-daemon[870]: time=2026-09-23T05:12:57.884Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)" lb # [ 45.362760] niks3-server[967]: 2026/09/23 05:12:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign worker2 # [ 45.471487] nix-grpc-daemon[870]: time=2026-09-23T05:12:57.924Z level=INFO msg="Uploading 1 narinfos" lb # [ 45.377229] niks3-server[967]: 2026/09/23 05:12:57 INFO Signed narinfos id=5 count=1 lb # [ 45.384700] niks3-server[967]: 2026/09/23 05:12:57 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls worker2 # [ 45.499433] nix-grpc-daemon[870]: time=2026-09-23T05:12:57.952Z level=INFO msg="Upload complete. (250ms)" lb # [ 45.395957] niks3-server[967]: 2026/09/23 05:12:57 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst worker2 # [ 45.508374] nix-grpc-daemon[769]: ts=2026-09-23T05:12:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 45.410427] niks3-server[967]: 2026/09/23 05:12:57 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete lb # [ 45.414829] niks3-server[967]: 2026/09/23 05:12:57 INFO Completed upload id=5 lb # [ 45.417703] niks3-server[967]: 2026/09/23 05:12:57 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo worker2 # [ 45.527444] nix-grpc-daemon[769]: ts=2026-09-23T05:12:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 45.534353] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 45.539820] nix-daemon[855]: reaped child process 947, status = succeeded client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051... client # farm-top> worker2: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker2 # [ 45.555760] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1 worker2 # [ 46.870186] nix-daemon[855]: reaped child process 956, status = succeeded worker2 # [ 46.938391] nix-daemon[855]: accepted connection from pid 972, user nix-grpc-daemon (trusted) worker2 # [ 46.960369] nix-daemon[855]: reaped child process 975, status = succeeded lb # [ 46.870351] niks3-server[967]: 2026/09/23 05:12:59 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 46.982986] nix-grpc-daemon[870]: time=2026-09-23T05:12:59.433Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 46.984866] nix-grpc-daemon[870]: time=2026-09-23T05:12:59.434Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)" lb # [ 46.922664] niks3-server[967]: 2026/09/23 05:12:59 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls lb # [ 46.935986] niks3-server[967]: 2026/09/23 05:12:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign worker2 # [ 47.043203] nix-grpc-daemon[870]: time=2026-09-23T05:12:59.495Z level=INFO msg="Uploading 1 narinfos" lb # [ 46.947947] niks3-server[967]: 2026/09/23 05:12:59 INFO Signed narinfos id=6 count=1 lb # [ 46.955415] niks3-server[967]: 2026/09/23 05:12:59 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv lb # [ 46.965151] niks3-server[967]: 2026/09/23 05:12:59 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst lb # [ 46.978521] niks3-server[967]: 2026/09/23 05:12:59 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete lb # [ 46.986205] niks3-server[967]: 2026/09/23 05:12:59 INFO Completed upload id=6 worker2 # [ 47.088162] nix-grpc-daemon[870]: time=2026-09-23T05:12:59.541Z level=INFO msg="Upload complete. (320ms)" lb # [ 46.989760] niks3-server[967]: 2026/09/23 05:12:59 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo worker2 # [ 47.102429] nix-grpc-daemon[769]: ts=2026-09-23T05:12:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=1 worker2 # [ 47.125578] nix-daemon[855]: reaped child process 954, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 7.12 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.20 seconds) client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.03 seconds) client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client # copying 3 paths... client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'... client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'... client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.27 seconds) (finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.72 seconds) subtest: repeat is answered Cached by the scheduler without building worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker1 # finding garbage collector roots... worker1 # deleting specified paths... worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker1 # deleting unused links... worker1 # note: hard linking is currently saving 0.0 KiB worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.25 seconds) worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker2 # finding garbage collector roots... worker2 # deleting specified paths... worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker2 # deleting unused links... worker2 # note: hard linking is currently saving 0.0 KiB worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.23 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 worker2 # [ 48.474881] nix-grpc-daemon[769]: ts=2026-09-23T05:13:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:58726 client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked): client # /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker1 # [ 48.433355] nix-grpc-daemon[766]: ts=2026-09-23T05:13:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 worker2 # [ 48.538817] nix-grpc-daemon[769]: ts=2026-09-23T05:13:00Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=1 client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 0.34 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.17 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: repeat is answered Cached by the scheduler without building, in 1.28 seconds) subtest: inputs already in the cache are not sent to the scheduler lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.07 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2 lb # [ 49.184670] envoy[895]: conn peer=[2001:db8:1::1]:46312 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=94 lb # [ 49.189558] envoy[895]: conn peer=[2001:db8:1::1]:46324 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=20 lb # [ 49.194236] envoy[895]: conn peer=[2001:db8:1::1]:46334 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=19 lb # [ 49.199271] envoy[895]: conn peer=[2001:db8:1::1]:46348 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9549 tx=4393 ms=6919 lb # [ 49.204404] envoy[895]: conn peer=[2001:db8:1::1]:34886 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=480 ms=88 client # don't know how to build these paths: client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker1 # [ 49.376737] nix-grpc-daemon[766]: ts=2026-09-23T05:13:01Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:49090 worker2 # [ 49.452585] nix-grpc-daemon[769]: ts=2026-09-23T05:13:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 49.464821] nix-grpc-daemon[870]: time=2026-09-23T05:13:01.916Z level=INFO msg="All 2 paths already cached" worker2 # [ 49.485627] nix-grpc-daemon[769]: ts=2026-09-23T05:13:01Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=0 worker2 # [ 49.491372] nix-grpc-daemon[769]: ts=2026-09-23T05:13:01Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 49.531785] nix-grpc-daemon[769]: ts=2026-09-23T05:13:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 49.548147] nix-grpc-daemon[769]: ts=2026-09-23T05:13:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 49.560096] nix-grpc-daemon[870]: time=2026-09-23T05:13:02.013Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'... worker2 # [ 49.569944] nix-grpc-daemon[769]: ts=2026-09-23T05:13:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 49.586560] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 49.754564] nix-daemon[855]: accepted connection from pid 1004, user nix-grpc-daemon (trusted) worker2 # [ 49.778485] nix-daemon[855]: reaped child process 1007, status = succeeded lb # [ 49.677831] niks3-server[967]: 2026/09/23 05:13:02 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 49.790589] nix-grpc-daemon[870]: time=2026-09-23T05:13:02.243Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 49.792370] nix-grpc-daemon[870]: time=2026-09-23T05:13:02.243Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)" lb # [ 49.717667] niks3-server[967]: 2026/09/23 05:13:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign worker2 # [ 49.826676] nix-grpc-daemon[870]: time=2026-09-23T05:13:02.279Z level=INFO msg="Uploading 1 narinfos" lb # [ 49.732741] niks3-server[967]: 2026/09/23 05:13:02 INFO Signed narinfos id=7 count=1 lb # [ 49.741506] niks3-server[967]: 2026/09/23 05:13:02 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst lb # [ 49.750697] niks3-server[967]: 2026/09/23 05:13:02 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls lb # [ 49.759986] niks3-server[967]: 2026/09/23 05:13:02 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete worker2 # [ 49.867630] nix-grpc-daemon[870]: time=2026-09-23T05:13:02.320Z level=INFO msg="Upload complete. (259ms)" lb # [ 49.766823] niks3-server[967]: 2026/09/23 05:13:02 INFO Completed upload id=7 lb # [ 49.770303] niks3-server[967]: 2026/09/23 05:13:02 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo worker2 # [ 49.877728] nix-grpc-daemon[769]: ts=2026-09-23T05:13:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=0 paths=1 nar_bytes_in=752 worker2 # [ 49.893151] nix-grpc-daemon[769]: ts=2026-09-23T05:13:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 49.902738] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 49.916341] nix-daemon[855]: reaped child process 1002, status = succeeded client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051... client # farm-top> worker2: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker2 # [ 49.931258] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1b worker2 # [ 51.220621] nix-daemon[855]: reaped child process 1011, status = succeeded worker2 # [ 51.329018] nix-daemon[855]: accepted connection from pid 1027, user nix-grpc-daemon (trusted) worker2 # [ 51.352569] nix-daemon[855]: reaped child process 1030, status = succeeded lb # [ 51.249682] niks3-server[967]: 2026/09/23 05:13:03 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 51.364509] nix-grpc-daemon[870]: time=2026-09-23T05:13:03.814Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 51.366135] nix-grpc-daemon[870]: time=2026-09-23T05:13:03.814Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)" lb # [ 51.295718] niks3-server[967]: 2026/09/23 05:13:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign worker2 # [ 51.409375] nix-grpc-daemon[870]: time=2026-09-23T05:13:03.862Z level=INFO msg="Uploading 1 narinfos" lb # [ 51.313340] niks3-server[967]: 2026/09/23 05:13:03 INFO Signed narinfos id=8 count=1 lb # [ 51.315759] niks3-server[967]: 2026/09/23 05:13:03 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv lb # [ 51.329342] niks3-server[967]: 2026/09/23 05:13:03 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls lb # [ 51.333578] niks3-server[967]: 2026/09/23 05:13:03 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst lb # [ 51.344433] niks3-server[967]: 2026/09/23 05:13:03 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete lb # [ 51.348834] niks3-server[967]: 2026/09/23 05:13:03 INFO Completed upload id=8 lb # [ 51.354848] niks3-server[967]: 2026/09/23 05:13:03 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo worker2 # [ 51.445467] nix-grpc-daemon[870]: time=2026-09-23T05:13:03.898Z level=INFO msg="Upload complete. (295ms)" worker2 # [ 51.461685] nix-grpc-daemon[769]: ts=2026-09-23T05:13:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=1 worker2 # [ 51.480662] nix-daemon[855]: reaped child process 1009, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.29 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.20 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.07 seconds) (finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.85 seconds) subtest: outputs read back through any worker carry the cache signature client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2 client # don't know how to build these paths: client # /nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker2 # [ 52.107683] nix-grpc-daemon[769]: ts=2026-09-23T05:13:04Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:58726 worker1 # [ 52.045651] nix-grpc-daemon[766]: ts=2026-09-23T05:13:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 worker2 # [ 52.141132] nix-grpc-daemon[769]: ts=2026-09-23T05:13:04Z level=debug event=want client=4 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv cached=0 worker2 # [ 52.147542] nix-grpc-daemon[769]: ts=2026-09-23T05:13:04Z level=debug event=assigned drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 52.081527] nix-grpc-daemon[766]: ts=2026-09-23T05:13:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 52.099054] nix-grpc-daemon[766]: ts=2026-09-23T05:13:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' to 'lb:50051'... worker1 # [ 52.116794] nix-grpc-daemon[766]: ts=2026-09-23T05:13:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 52.133511] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 52.236631] nix-grpc-daemon[766]: ts=2026-09-23T05:13:04Z level=info event=niks3_push_started pid=860 worker1 # [ 52.441737] nix-grpc-daemon[860]: time=2026-09-23T05:13:04.947Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 52.560564] nix-daemon[832]: accepted connection from pid 873, user nix-grpc-daemon (trusted) worker1 # [ 52.581370] nix-daemon[832]: reaped child process 876, status = succeeded lb # [ 52.557933] niks3-server[967]: 2026/09/23 05:13:05 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 52.600932] nix-grpc-daemon[860]: time=2026-09-23T05:13:05.106Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 52.602517] nix-grpc-daemon[860]: time=2026-09-23T05:13:05.106Z level=INFO msg="Uploading 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv (512B)" lb # [ 52.606694] niks3-server[967]: 2026/09/23 05:13:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign worker1 # [ 52.648093] nix-grpc-daemon[860]: time=2026-09-23T05:13:05.153Z level=INFO msg="Uploading 1 narinfos" lb # [ 52.616838] niks3-server[967]: 2026/09/23 05:13:05 INFO Signed narinfos id=9 count=1 lb # [ 52.628279] niks3-server[967]: 2026/09/23 05:13:05 INFO Registered completed upload object_key=nar/13mhmmy2qay13imbbdi287vhryz1hwnyg0w6agd9s0ph37wnsqll.nar.zst lb # [ 52.640192] niks3-server[967]: 2026/09/23 05:13:05 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.ls lb # [ 52.652986] niks3-server[967]: 2026/09/23 05:13:05 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete lb # [ 52.660657] niks3-server[967]: 2026/09/23 05:13:05 INFO Completed upload id=9 worker1 # [ 52.690989] nix-grpc-daemon[860]: time=2026-09-23T05:13:05.197Z level=INFO msg="Upload complete. (245ms)" lb # [ 52.665766] niks3-server[967]: 2026/09/23 05:13:05 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.narinfo worker1 # [ 52.704779] nix-grpc-daemon[766]: ts=2026-09-23T05:13:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 52.719478] nix-grpc-daemon[766]: ts=2026-09-23T05:13:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 52.726585] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 52.731590] nix-daemon[832]: reaped child process 858, status = succeeded client # building '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' on lb:50051... worker1 # [ 52.743770] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) client # farm-a> node-a: building 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-sig worker1 # [ 54.177946] nix-daemon[832]: reaped child process 880, status = succeeded worker1 # [ 54.270674] nix-daemon[832]: accepted connection from pid 896, user nix-grpc-daemon (trusted) worker1 # [ 54.291163] nix-daemon[832]: reaped child process 899, status = succeeded lb # [ 54.267926] niks3-server[967]: 2026/09/23 05:13:06 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 54.312572] nix-grpc-daemon[860]: time=2026-09-23T05:13:06.816Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 54.314091] nix-grpc-daemon[860]: time=2026-09-23T05:13:06.816Z level=INFO msg="Uploading 2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a (128B)" lb # [ 54.316472] niks3-server[967]: 2026/09/23 05:13:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign worker1 # [ 54.356397] nix-grpc-daemon[860]: time=2026-09-23T05:13:06.862Z level=INFO msg="Uploading 1 narinfos" lb # [ 54.326043] niks3-server[967]: 2026/09/23 05:13:06 INFO Signed narinfos id=10 count=1 lb # [ 54.339070] niks3-server[967]: 2026/09/23 05:13:06 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete lb # [ 54.367552] niks3-server[967]: 2026/09/23 05:13:06 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst lb # [ 54.377640] niks3-server[967]: 2026/09/23 05:13:06 INFO Registered completed upload object_key=log/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv lb # [ 54.391226] niks3-server[967]: 2026/09/23 05:13:06 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.ls worker1 # [ 54.429312] nix-grpc-daemon[860]: time=2026-09-23T05:13:06.935Z level=INFO msg="Upload complete. (272ms)" lb # [ 54.400945] niks3-server[967]: 2026/09/23 05:13:06 INFO Completed upload id=10 lb # [ 54.407735] niks3-server[967]: 2026/09/23 05:13:06 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.narinfo worker1 # [ 54.441351] nix-grpc-daemon[766]: ts=2026-09-23T05:13:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=1 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv assign_id=5 outputs=1 worker2 # [ 54.519623] nix-grpc-daemon[769]: ts=2026-09-23T05:13:06Z level=debug event=want client=4 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv cached=0 worker2 # [ 54.525755] nix-grpc-daemon[769]: ts=2026-09-23T05:13:06Z level=debug event=assigned drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 54.464638] nix-daemon[832]: reaped child process 878, status = succeeded worker1 # [ 54.497015] nix-grpc-daemon[766]: ts=2026-09-23T05:13:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 54.513897] nix-grpc-daemon[766]: ts=2026-09-23T05:13:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 54.530230] nix-grpc-daemon[860]: time=2026-09-23T05:13:07.036Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' to 'lb:50051'... worker1 # [ 54.541430] nix-grpc-daemon[766]: ts=2026-09-23T05:13:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 54.555212] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 54.719598] nix-daemon[832]: accepted connection from pid 903, user nix-grpc-daemon (trusted) worker1 # [ 54.742634] nix-daemon[832]: reaped child process 906, status = succeeded lb # [ 54.708390] niks3-server[967]: 2026/09/23 05:13:07 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 54.749753] nix-grpc-daemon[860]: time=2026-09-23T05:13:07.255Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 54.751408] nix-grpc-daemon[860]: time=2026-09-23T05:13:07.256Z level=INFO msg="Uploading z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv (624B)" lb # [ 54.739913] niks3-server[967]: 2026/09/23 05:13:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign worker1 # [ 54.779006] nix-grpc-daemon[860]: time=2026-09-23T05:13:07.285Z level=INFO msg="Uploading 1 narinfos" lb # [ 54.753389] niks3-server[967]: 2026/09/23 05:13:07 INFO Signed narinfos id=11 count=1 lb # [ 54.760736] niks3-server[967]: 2026/09/23 05:13:07 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.ls lb # [ 54.777060] niks3-server[967]: 2026/09/23 05:13:07 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete worker1 # [ 54.815494] nix-grpc-daemon[860]: time=2026-09-23T05:13:07.322Z level=INFO msg="Upload complete. (243ms)" lb # [ 54.787620] niks3-server[967]: 2026/09/23 05:13:07 INFO Registered completed upload object_key=nar/0cnhkgsbmxv5hj7zw3fa59qfxikw6251mlhz2gz1v6pgn8c8shwp.nar.zst lb # [ 54.793170] niks3-server[967]: 2026/09/23 05:13:07 INFO Completed upload id=11 worker1 # [ 54.824740] nix-grpc-daemon[766]: ts=2026-09-23T05:13:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 54.796550] niks3-server[967]: 2026/09/23 05:13:07 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.narinfo worker1 # [ 54.842223] nix-grpc-daemon[766]: ts=2026-09-23T05:13:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 54.849242] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 54.854926] nix-daemon[832]: reaped child process 901, status = succeeded client # building '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' on lb:50051... worker1 # [ 54.867816] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) client # farm-b> node-a: building z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-sig worker1 # [ 55.960157] nix-daemon[832]: reaped child process 910, status = succeeded worker1 # [ 56.059922] nix-daemon[832]: accepted connection from pid 926, user nix-grpc-daemon (trusted) worker1 # [ 56.081701] nix-daemon[832]: reaped child process 929, status = succeeded lb # [ 56.051922] niks3-server[967]: 2026/09/23 05:13:08 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 56.095967] nix-grpc-daemon[860]: time=2026-09-23T05:13:08.601Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 56.097875] nix-grpc-daemon[860]: time=2026-09-23T05:13:08.601Z level=INFO msg="Uploading h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b (176B)" lb # [ 56.093362] niks3-server[967]: 2026/09/23 05:13:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign worker1 # [ 56.136549] nix-grpc-daemon[860]: time=2026-09-23T05:13:08.642Z level=INFO msg="Uploading 1 narinfos" lb # [ 56.108326] niks3-server[967]: 2026/09/23 05:13:08 INFO Signed narinfos id=12 count=1 lb # [ 56.114417] niks3-server[967]: 2026/09/23 05:13:08 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.ls lb # [ 56.124113] niks3-server[967]: 2026/09/23 05:13:08 INFO Registered completed upload object_key=log/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv lb # [ 56.131961] niks3-server[967]: 2026/09/23 05:13:08 INFO Registered completed upload object_key=nar/0f75gpfkjj0g9xi2113yvvpdw0l16spc2f1xik1ifnzfbrvx4pfk.nar.zst lb # [ 56.142589] niks3-server[967]: 2026/09/23 05:13:08 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete lb # [ 56.146912] niks3-server[967]: 2026/09/23 05:13:08 INFO Completed upload id=12 worker1 # [ 56.172648] nix-grpc-daemon[860]: time=2026-09-23T05:13:08.679Z level=INFO msg="Upload complete. (287ms)" lb # [ 56.151108] niks3-server[967]: 2026/09/23 05:13:08 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.narinfo worker1 # [ 56.185585] nix-grpc-daemon[766]: ts=2026-09-23T05:13:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=1 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv assign_id=6 outputs=1 worker2 # [ 56.265996] nix-grpc-daemon[769]: ts=2026-09-23T05:13:08Z level=debug event=want client=4 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv cached=0 worker2 # [ 56.266303] nix-grpc-daemon[769]: ts=2026-09-23T05:13:08Z level=debug event=assigned drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 56.207687] nix-daemon[832]: reaped child process 908, status = succeeded worker1 # [ 56.240637] nix-grpc-daemon[766]: ts=2026-09-23T05:13:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 56.256657] nix-grpc-daemon[766]: ts=2026-09-23T05:13:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 56.271664] nix-grpc-daemon[860]: time=2026-09-23T05:13:08.778Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' to 'lb:50051'... worker1 # [ 56.283590] nix-grpc-daemon[766]: ts=2026-09-23T05:13:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 56.298087] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 56.456123] nix-daemon[832]: accepted connection from pid 933, user nix-grpc-daemon (trusted) worker1 # [ 56.478161] nix-daemon[832]: reaped child process 936, status = succeeded lb # [ 56.449511] niks3-server[967]: 2026/09/23 05:13:08 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 56.491066] nix-grpc-daemon[860]: time=2026-09-23T05:13:08.997Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 56.493252] nix-grpc-daemon[860]: time=2026-09-23T05:13:08.997Z level=INFO msg="Uploading knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv (752B)" lb # [ 56.481374] niks3-server[967]: 2026/09/23 05:13:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign worker1 # [ 56.520353] nix-grpc-daemon[860]: time=2026-09-23T05:13:09.026Z level=INFO msg="Uploading 1 narinfos" lb # [ 56.495582] niks3-server[967]: 2026/09/23 05:13:08 INFO Signed narinfos id=13 count=1 lb # [ 56.506492] niks3-server[967]: 2026/09/23 05:13:09 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.ls worker1 # [ 56.550582] nix-grpc-daemon[860]: time=2026-09-23T05:13:09.057Z level=INFO msg="Upload complete. (235ms)" lb # [ 56.517639] niks3-server[967]: 2026/09/23 05:13:09 INFO Registered completed upload object_key=nar/0lpagyrngav67x80d1hq0njjlkmqdkvqawraz8dsp36ag50v6iwf.nar.zst lb # [ 56.526198] niks3-server[967]: 2026/09/23 05:13:09 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete worker1 # [ 56.559988] nix-grpc-daemon[766]: ts=2026-09-23T05:13:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 56.533249] niks3-server[967]: 2026/09/23 05:13:09 INFO Completed upload id=13 lb # [ 56.538134] niks3-server[967]: 2026/09/23 05:13:09 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.narinfo worker1 # [ 56.576841] nix-grpc-daemon[766]: ts=2026-09-23T05:13:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 56.583106] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 56.588872] nix-daemon[832]: reaped child process 931, status = succeeded client # building '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' on lb:50051... client # farm-top> node-a: building knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker1 # [ 56.602950] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-sig worker1 # [ 57.470124] nix-daemon[832]: reaped child process 940, status = succeeded worker1 # [ 57.605385] nix-daemon[832]: accepted connection from pid 956, user nix-grpc-daemon (trusted) worker1 # [ 57.629065] nix-daemon[832]: reaped child process 959, status = succeeded lb # [ 57.595343] niks3-server[967]: 2026/09/23 05:13:10 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 57.639988] nix-grpc-daemon[860]: time=2026-09-23T05:13:10.144Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 57.641753] nix-grpc-daemon[860]: time=2026-09-23T05:13:10.144Z level=INFO msg="Uploading 4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top (232B)" lb # [ 57.642862] niks3-server[967]: 2026/09/23 05:13:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign worker1 # [ 57.683439] nix-grpc-daemon[860]: time=2026-09-23T05:13:10.189Z level=INFO msg="Uploading 1 narinfos" lb # [ 57.656621] niks3-server[967]: 2026/09/23 05:13:10 INFO Signed narinfos id=14 count=1 lb # [ 57.663712] niks3-server[967]: 2026/09/23 05:13:10 INFO Registered completed upload object_key=nar/07ar364rwqqniwi2nkdpqlgv7ws7702n5lqz703ackfmssrh40v8.nar.zst lb # [ 57.672990] niks3-server[967]: 2026/09/23 05:13:10 INFO Registered completed upload object_key=log/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv lb # [ 57.683137] niks3-server[967]: 2026/09/23 05:13:10 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete lb # [ 57.690990] niks3-server[967]: 2026/09/23 05:13:10 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.ls worker1 # [ 57.722942] nix-grpc-daemon[860]: time=2026-09-23T05:13:10.229Z level=INFO msg="Upload complete. (308ms)" lb # [ 57.696237] niks3-server[967]: 2026/09/23 05:13:10 INFO Completed upload id=14 lb # [ 57.699439] niks3-server[967]: 2026/09/23 05:13:10 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.narinfo worker1 # [ 57.737228] nix-grpc-daemon[766]: ts=2026-09-23T05:13:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=1 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv assign_id=7 outputs=1 worker1 # [ 57.755854] nix-daemon[832]: reaped child process 938, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2, in 5.97 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath, in 0.19 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker1 # [ 58.107836] nix-grpc-daemon[766]: ts=2026-09-23T05:13:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:53928 worker1 # [ 58.150632] nix-grpc-daemon[766]: ts=2026-09-23T05:13:10Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:53928 client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.25 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker2 # [ 58.473239] nix-grpc-daemon[769]: ts=2026-09-23T05:13:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:34360 client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked): client # /nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a client # /nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b client # /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker2 # [ 58.517029] nix-grpc-daemon[769]: ts=2026-09-23T05:13:10Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:34360 worker2 # [ 58.526335] nix-grpc-daemon[769]: copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'http://lb:5751'... worker2 # [ 58.546054] nix-grpc-daemon[769]: copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'http://lb:5751'... worker2 # [ 58.561060] nix-grpc-daemon[769]: copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.32 seconds) client: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top client # finding garbage collector roots... client # removing stale temporary roots file "/nix/var/nix/temproots/919" client # removing stale temporary roots file "/nix/var/nix/temproots/942" client # removing stale temporary roots file "/nix/var/nix/temproots/906" client # removing stale temporary roots file "/nix/var/nix/temproots/927" client # removing stale temporary roots file "/nix/var/nix/temproots/866" client # removing stale temporary roots file "/nix/var/nix/temproots/887" client # removing stale temporary roots file "/nix/var/nix/temproots/849" client # removing stale temporary roots file "/nix/var/nix/temproots/898" client # deleting specified paths... client # deleting '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' client # deleting unused links... client # note: hard linking is currently saving 0.0 KiB worker2 # [ 58.973380] nix-grpc-daemon[769]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:58726 worker1 # [ 58.908963] nix-grpc-daemon[766]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 58.921876] nix-grpc-daemon[860]: time=2026-09-23T05:13:11.427Z level=INFO msg="All 1 paths already cached" worker1 # [ 58.931152] nix-grpc-daemon[766]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 58.940750] nix-grpc-daemon[766]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 58.950096] nix-grpc-daemon[766]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49090 client # copying 3 paths... worker1 # [ 58.964998] nix-grpc-daemon[766]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49090 client # copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'grpc://lb:50051'... worker2 # [ 59.063949] nix-grpc-daemon[769]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 59.071776] nix-grpc-daemon[769]: ts=2026-09-23T05:13:11Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=0 paths=1 nar_bytes_out=232 worker2 # [ 59.081607] nix-grpc-daemon[769]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 59.088701] nix-grpc-daemon[769]: ts=2026-09-23T05:13:11Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=0 paths=1 nar_bytes_out=176 worker1 # [ 59.028262] nix-grpc-daemon[766]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 59.035923] nix-grpc-daemon[766]: ts=2026-09-23T05:13:11Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=0 paths=1 nar_bytes_out=128 client # copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'grpc://lb:50051'... client # copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'grpc://lb:50051'... client: (finished: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.57 seconds) (finished: subtest: outputs read back through any worker carry the cache signature, in 7.30 seconds) subtest: two clients wanting the same drv build it once client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup client # [ 59.184971] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup1.service; invocation ID: c839abcf7c834d46ae1352b88ca67d2a client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.07 seconds) client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup lb # [ 59.186708] envoy[895]: conn peer=[2001:db8:1::1]:34888 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4185 tx=1920 ms=2030 lb # [ 59.191725] envoy[895]: conn peer=[2001:db8:1::1]:36950 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9528 tx=4368 ms=5716 lb # [ 59.197004] envoy[895]: conn peer=[2001:db8:1::1]:36964 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1108 tx=2928 ms=186 lb # [ 59.201890] envoy[895]: conn peer=[2001:db8:1::1]:36990 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=329 ms=64 lb # [ 59.206740] envoy[895]: conn peer=[2001:db8:1::1]:36986 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=408 tx=375 ms=81 lb # [ 59.212455] envoy[895]: conn peer=[2001:db8:1::1]:36980 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=384 tx=409 ms=98 worker2 # [ 59.435690] nix-grpc-daemon[769]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:58726 client # [ 59.394816] nix[995]: don't know how to build these paths: client # [ 59.395972] nix[995]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker2 # [ 59.453557] nix-grpc-daemon[769]: ts=2026-09-23T05:13:11Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker2 # [ 59.461106] nix-grpc-daemon[769]: ts=2026-09-23T05:13:11Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 59.499024] nix-grpc-daemon[769]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 59.514484] nix-grpc-daemon[769]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58726 client # [ 59.480175] nix[995]: copying 1 paths to 'lb:50051'... worker2 # [ 59.530148] nix-grpc-daemon[769]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 client # [ 59.482616] nix[995]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'... worker2 # [ 59.548159] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 59.708456] nix-daemon[855]: accepted connection from pid 1050, user nix-grpc-daemon (trusted) worker2 # [ 59.730473] nix-daemon[855]: reaped child process 1053, status = succeeded lb # [ 59.638133] niks3-server[967]: 2026/09/23 05:13:12 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 59.749949] nix-grpc-daemon[870]: time=2026-09-23T05:13:12.202Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 59.752000] nix-grpc-daemon[870]: time=2026-09-23T05:13:12.202Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)" lb # [ 59.670924] niks3-server[967]: 2026/09/23 05:13:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign worker2 # [ 59.780473] nix-grpc-daemon[870]: time=2026-09-23T05:13:12.233Z level=INFO msg="Uploading 1 narinfos" lb # [ 59.683579] niks3-server[967]: 2026/09/23 05:13:12 INFO Signed narinfos id=15 count=1 lb # [ 59.693817] niks3-server[967]: 2026/09/23 05:13:12 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls lb # [ 59.704569] niks3-server[967]: 2026/09/23 05:13:12 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst lb # [ 59.722434] niks3-server[967]: 2026/09/23 05:13:12 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete lb # [ 59.731623] niks3-server[967]: 2026/09/23 05:13:12 INFO Completed upload id=15 worker2 # [ 59.831573] nix-grpc-daemon[870]: time=2026-09-23T05:13:12.284Z level=INFO msg="Upload complete. (266ms)" lb # [ 59.735031] niks3-server[967]: 2026/09/23 05:13:12 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo worker2 # [ 59.843586] nix-grpc-daemon[769]: ts=2026-09-23T05:13:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=0 paths=1 nar_bytes_in=432 worker2 # [ 59.865238] nix-grpc-daemon[769]: ts=2026-09-23T05:13:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 59.873841] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 59.880226] nix-daemon[855]: reaped child process 1048, status = succeeded client # [ 59.844066] nix[995]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... worker2 # [ 59.897968] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) client # [ 60.283740] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup2.service; invocation ID: 3b7b4e4a75144222baa020fc3930a659 client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.12 seconds) client # [ 60.535263] nix[1011]: don't know how to build these paths: client # [ 60.536535] nix[1011]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker1 # [ 60.513212] nix-grpc-daemon[766]: ts=2026-09-23T05:13:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:49090 worker2 # [ 60.591535] nix-grpc-daemon[769]: ts=2026-09-23T05:13:13Z level=debug event=want client=6 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker2 # [ 60.596715] nix-grpc-daemon[769]: ts=2026-09-23T05:13:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 client # [ 60.550043] nix[1011]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... client: must succeed: sleep 2 client: (finished: must succeed: sleep 2, in 2.06 seconds) client: waiting for success: ! systemctl is-active dedup1 dedup2 worker2 # [ 63.927416] nix-daemon[855]: reaped child process 1057, status = succeeded worker2 # [ 64.025289] nix-daemon[855]: accepted connection from pid 1089, user nix-grpc-daemon (trusted) worker2 # [ 64.047155] nix-daemon[855]: reaped child process 1092, status = succeeded lb # [ 63.951167] niks3-server[967]: 2026/09/23 05:13:16 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 64.064425] nix-grpc-daemon[870]: time=2026-09-23T05:13:16.516Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 64.065944] nix-grpc-daemon[870]: time=2026-09-23T05:13:16.516Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)" lb # [ 63.998835] niks3-server[967]: 2026/09/23 05:13:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign worker2 # [ 64.109734] nix-grpc-daemon[870]: time=2026-09-23T05:13:16.562Z level=INFO msg="Uploading 1 narinfos" lb # [ 64.012594] niks3-server[967]: 2026/09/23 05:13:16 INFO Signed narinfos id=16 count=1 lb # [ 64.015115] niks3-server[967]: 2026/09/23 05:13:16 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst lb # [ 64.034205] niks3-server[967]: 2026/09/23 05:13:16 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls lb # [ 64.037564] niks3-server[967]: 2026/09/23 05:13:16 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker2 # [ 64.142957] nix-grpc-daemon[870]: time=2026-09-23T05:13:16.596Z level=INFO msg="Upload complete. (230ms)" lb # [ 64.041848] niks3-server[967]: 2026/09/23 05:13:16 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete lb # [ 64.046334] niks3-server[967]: 2026/09/23 05:13:16 INFO Completed upload id=16 lb # [ 64.051296] niks3-server[967]: 2026/09/23 05:13:16 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo worker2 # [ 64.154099] nix-grpc-daemon[769]: ts=2026-09-23T05:13:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker2 # [ 64.174574] nix-grpc-daemon[769]: ts=2026-09-23T05:13:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker2 # [ 64.180607] nix-daemon[855]: reaped child process 1055, status = succeeded client # [ 64.152763] systemd[1]: dedup1.service: Deactivated successfully. client # [ 64.158413] systemd[1]: dedup1.service: Consumed 182ms CPU time over 4.966s wall clock time, 41.5M memory peak, 7.3K incoming IP traffic, 8.5K outgoing IP traffic. client # [ 64.201370] systemd[1]: dedup2.service: Deactivated successfully. client # [ 64.205256] systemd[1]: dedup2.service: Consumed 218ms CPU time over 3.921s wall clock time, 33.2M memory peak, 5.9K incoming IP traffic, 5.5K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.13 seconds) client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; } client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.06 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: two clients wanting the same drv build it once, in 6.16 seconds) subtest: --timeout and --max-silent-time reach the builder worker1 # [ 65.510687] nix-grpc-daemon[766]: ts=2026-09-23T05:13:18Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:49090 worker2 # [ 65.600875] nix-grpc-daemon[769]: ts=2026-09-23T05:13:18Z level=debug event=want client=7 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv cached=0 worker1 # [ 65.535421] nix-grpc-daemon[766]: ts=2026-09-23T05:13:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 65.555243] nix-grpc-daemon[766]: ts=2026-09-23T05:13:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 65.573042] nix-grpc-daemon[766]: ts=2026-09-23T05:13:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 65.589695] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 65.749980] nix-daemon[832]: accepted connection from pid 984, user nix-grpc-daemon (trusted) worker1 # [ 65.773254] nix-daemon[832]: reaped child process 987, status = succeeded lb # [ 65.739548] niks3-server[967]: 2026/09/23 05:13:18 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 65.816277] nix-grpc-daemon[860]: time=2026-09-23T05:13:18.321Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 65.819517] nix-grpc-daemon[860]: time=2026-09-23T05:13:18.321Z level=INFO msg="Uploading sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv (424B)" lb # [ 65.808471] niks3-server[967]: 2026/09/23 05:13:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign worker1 # [ 65.849914] nix-grpc-daemon[860]: time=2026-09-23T05:13:18.356Z level=INFO msg="Uploading 1 narinfos" lb # [ 65.823580] niks3-server[967]: 2026/09/23 05:13:18 INFO Signed narinfos id=17 count=1 lb # [ 65.831126] niks3-server[967]: 2026/09/23 05:13:18 INFO Registered completed upload object_key=nar/1m12xmj6yzmb9y2j8ly0yvh5fyzy1nqk7ajhqwr4wn3b5h0bsch5.nar.zst lb # [ 65.843599] niks3-server[967]: 2026/09/23 05:13:18 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.ls worker1 # [ 65.890570] nix-grpc-daemon[860]: time=2026-09-23T05:13:18.397Z level=INFO msg="Upload complete. (284ms)" lb # [ 65.856916] niks3-server[967]: 2026/09/23 05:13:18 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete lb # [ 65.866926] niks3-server[967]: 2026/09/23 05:13:18 INFO Completed upload id=17 worker1 # [ 65.900427] nix-grpc-daemon[766]: ts=2026-09-23T05:13:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=0 paths=1 nar_bytes_in=424 lb # [ 65.873350] niks3-server[967]: 2026/09/23 05:13:18 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.narinfo worker1 # [ 65.921058] nix-grpc-daemon[766]: ts=2026-09-23T05:13:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 65.932429] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 65.938954] nix-daemon[832]: reaped child process 982, status = succeeded worker1 # [ 65.952785] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker2 # [ 65.606666] nix-grpc-daemon[769]: ts=2026-09-23T05:13:18Z level=debug event=assigned drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv worker=192.168.1.3:50051 clients=1 lb # [ 69.186403] envoy[895]: conn peer=[2001:db8:1::1]:36996 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3018 tx=1564 ms=4743 lb # [ 69.191471] envoy[895]: conn peer=[2001:db8:1::1]:36998 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1194 ms=3650 worker1 # [ 71.774462] nix-grpc-daemon[766]: ts=2026-09-23T05:13:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=5 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv assign_id=9 outputs=0 worker1 # [ 71.804816] nix-daemon[832]: reaped child process 989, status = succeeded worker1 # [ 71.806980] nix-daemon[832]: reaped child process 991, status = succeeded worker2 # [ 72.086508] nix-grpc-daemon[769]: ts=2026-09-23T05:13:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 72.103955] nix-grpc-daemon[769]: ts=2026-09-23T05:13:24Z level=debug event=want client=8 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv cached=0 worker2 # [ 72.112771] nix-grpc-daemon[769]: ts=2026-09-23T05:13:24Z level=debug event=assigned drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 72.121229] nix-grpc-daemon[769]: ts=2026-09-23T05:13:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 72.133261] nix-grpc-daemon[769]: ts=2026-09-23T05:13:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 72.148476] nix-grpc-daemon[769]: ts=2026-09-23T05:13:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 72.165073] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 72.326223] nix-daemon[855]: accepted connection from pid 1109, user nix-grpc-daemon (trusted) worker2 # [ 72.351254] nix-daemon[855]: reaped child process 1112, status = succeeded lb # [ 72.250110] niks3-server[967]: 2026/09/23 05:13:24 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 72.363976] nix-grpc-daemon[870]: time=2026-09-23T05:13:24.816Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 72.365657] nix-grpc-daemon[870]: time=2026-09-23T05:13:24.816Z level=INFO msg="Uploading pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv (424B)" lb # [ 72.293491] niks3-server[967]: 2026/09/23 05:13:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign worker2 # [ 72.403099] nix-grpc-daemon[870]: time=2026-09-23T05:13:24.856Z level=INFO msg="Uploading 1 narinfos" lb # [ 72.306920] niks3-server[967]: 2026/09/23 05:13:24 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.ls lb # [ 72.316249] niks3-server[967]: 2026/09/23 05:13:24 INFO Signed narinfos id=18 count=1 lb # [ 72.324355] niks3-server[967]: 2026/09/23 05:13:24 INFO Registered completed upload object_key=nar/14f5yrnllfav3svmw4v4svw15r4zlf5yv1p972jy14mmq6gdw943.nar.zst worker2 # [ 72.432780] nix-grpc-daemon[870]: time=2026-09-23T05:13:24.885Z level=INFO msg="Upload complete. (251ms)" lb # [ 72.332497] niks3-server[967]: 2026/09/23 05:13:24 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete lb # [ 72.337245] niks3-server[967]: 2026/09/23 05:13:24 INFO Completed upload id=18 lb # [ 72.340527] niks3-server[967]: 2026/09/23 05:13:24 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.narinfo worker2 # [ 72.443227] nix-grpc-daemon[769]: ts=2026-09-23T05:13:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 72.460382] nix-grpc-daemon[769]: ts=2026-09-23T05:13:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 72.467417] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 72.474071] nix-daemon[855]: reaped child process 1107, status = succeeded worker2 # [ 72.487563] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 78.312541] nix-grpc-daemon[769]: ts=2026-09-23T05:13:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=5 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv assign_id=10 outputs=0 worker2 # [ 78.351703] nix-daemon[855]: reaped child process 1114, status = succeeded worker2 # [ 78.356740] nix-daemon[855]: reaped child process 1116, status = succeeded (finished: subtest: --timeout and --max-silent-time reach the builder, in 13.20 seconds) subtest: the first client leaving does not fail the second client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share client # [ 78.567078] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share1.service; invocation ID: 48840f2fa3af4164b2cbec7ccc125278 client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 0.09 seconds) client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share worker2 # [ 78.827693] nix-grpc-daemon[769]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:58726 client # [ 78.785252] nix[1084]: don't know how to build these paths: client # [ 78.786212] nix[1084]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker2 # [ 78.848707] nix-grpc-daemon[769]: ts=2026-09-23T05:13:31Z level=debug event=want client=9 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 worker1 # [ 78.783563] nix-grpc-daemon[766]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker2 # [ 78.855820] nix-grpc-daemon[769]: ts=2026-09-23T05:13:31Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 78.799990] nix-grpc-daemon[766]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 client # [ 78.831939] nix[1084]: copying 1 paths to 'lb:50051'... client # [ 78.834705] nix[1084]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'... worker1 # [ 78.816193] nix-grpc-daemon[766]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 78.831641] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 78.990888] nix-daemon[832]: accepted connection from pid 1014, user nix-grpc-daemon (trusted) worker1 # [ 79.013822] nix-daemon[832]: reaped child process 1017, status = succeeded lb # [ 78.981087] niks3-server[967]: 2026/09/23 05:13:31 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 79.023527] nix-grpc-daemon[860]: time=2026-09-23T05:13:31.529Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 79.025780] nix-grpc-daemon[860]: time=2026-09-23T05:13:31.529Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)" lb # [ 79.014678] niks3-server[967]: 2026/09/23 05:13:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign worker1 # [ 79.056055] nix-grpc-daemon[860]: time=2026-09-23T05:13:31.562Z level=INFO msg="Uploading 1 narinfos" lb # [ 79.026818] niks3-server[967]: 2026/09/23 05:13:31 INFO Signed narinfos id=19 count=1 lb # [ 79.036777] niks3-server[967]: 2026/09/23 05:13:31 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst lb # [ 79.046606] niks3-server[967]: 2026/09/23 05:13:31 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls lb # [ 79.056387] niks3-server[967]: 2026/09/23 05:13:31 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete lb # [ 79.061825] niks3-server[967]: 2026/09/23 05:13:31 INFO Completed upload id=19 worker1 # [ 79.092501] nix-grpc-daemon[860]: time=2026-09-23T05:13:31.599Z level=INFO msg="Upload complete. (243ms)" lb # [ 79.066898] niks3-server[967]: 2026/09/23 05:13:31 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo worker1 # [ 79.104004] nix-grpc-daemon[766]: ts=2026-09-23T05:13:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=0 paths=1 nar_bytes_in=432 worker1 # [ 79.116641] nix-grpc-daemon[766]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 79.130131] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 79.138423] nix-daemon[832]: reaped child process 1012, status = succeeded worker1 # [ 79.147743] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) client # [ 79.166844] nix[1084]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... lb # [ 79.188790] envoy[895]: conn peer=[2001:db8:1::1]:44346 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2975 tx=977 ms=6289 lb # [ 79.194180] envoy[895]: conn peer=[2001:db8:1::1]:48532 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2972 tx=936 ms=6290 client # [ 79.671908] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share2.service; invocation ID: 14f6376761fd4fb795a4dca747ca819b client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 1.12 seconds) worker2 # [ 79.914576] nix-grpc-daemon[769]: ts=2026-09-23T05:13:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:58726 client # [ 79.871590] nix[1100]: don't know how to build these paths: client # [ 79.873667] nix[1100]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker2 # [ 79.933424] nix-grpc-daemon[769]: ts=2026-09-23T05:13:32Z level=debug event=want client=10 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 worker1 # [ 79.868654] nix-grpc-daemon[766]: ts=2026-09-23T05:13:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 client # [ 79.895621] nix[1100]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... client: must succeed: sleep 2; systemctl kill -s INT share1 client # [ 83.051567] systemd[1]: share1.service: Sent signal SIGINT to main process 1084 (nix) on client request. client # [ 83.061356] nix[1084]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.09 seconds) client: waiting for success: ! systemctl is-active share1 client # [ 83.070907] nix[1084]: [mutex.cc : 451] RAW: Unlock 0xf489edbfc788 @ 0xf48df22359a4 0xf48df330d40c 0xf48e127757bc 0xf48e12777414 0xf48e1276e1d4 0xf48e117b4f7c 0xf48e11542d80 client # [ 83.100992] systemd-coredump[1115]: Process 1084 (nix) of user 0 terminated abnormally with signal 11/SEGV, processing... client # [ 83.128175] systemd[1]: Created slice Slice /system/systemd-coredump. client # [ 83.141496] systemd[1]: Started Process Core Dump (PID 1115/UID 0). client # [ 85.602993] systemd-coredump[1120]: Cannot store coredump of 1084 (nix): No space left on device client # [ 85.743890] systemd-coredump[1120]: Process 1084 (nix) of user 0 terminated abnormally without generating a coredump. client # [ 85.762416] systemd[1]: systemd-coredump@0-1-1115_1116-0.service: Deactivated successfully. client # [ 85.766289] systemd[1]: systemd-coredump@0-1-1115_1116-0.service: Consumed 2.081s CPU time over 2.615s wall clock time, 968.8M memory peak, 926.3M written to disk. client # [ 85.775216] systemd[1]: share1.service: Main process exited, code=dumped, status=11/SEGV client # [ 85.778064] systemd[1]: share1.service: Failed with result 'core-dump'. client # [ 85.779311] systemd[1]: share1.service: Consumed 374ms CPU time over 7.208s wall clock time, 41.5M memory peak, 4K written to disk, 6.3K incoming IP traffic, 8.1K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share1, in 3.31 seconds) client: waiting for success: ! systemctl is-active share2 worker1 # [ 86.845069] nix-daemon[832]: reaped child process 1021, status = succeeded worker1 # [ 86.964211] nix-daemon[832]: accepted connection from pid 1055, user nix-grpc-daemon (trusted) worker1 # [ 86.989161] nix-daemon[832]: reaped child process 1058, status = succeeded lb # [ 86.963764] niks3-server[967]: 2026/09/23 05:13:39 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 87.008483] nix-grpc-daemon[860]: time=2026-09-23T05:13:39.511Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 87.010214] nix-grpc-daemon[860]: time=2026-09-23T05:13:39.511Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)" lb # [ 87.017263] niks3-server[967]: 2026/09/23 05:13:39 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv lb # [ 87.028718] niks3-server[967]: 2026/09/23 05:13:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign worker1 # [ 87.068142] nix-grpc-daemon[860]: time=2026-09-23T05:13:39.573Z level=INFO msg="Uploading 1 narinfos" lb # [ 87.039353] niks3-server[967]: 2026/09/23 05:13:39 INFO Signed narinfos id=20 count=1 lb # [ 87.048393] niks3-server[967]: 2026/09/23 05:13:39 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls lb # [ 87.056772] niks3-server[967]: 2026/09/23 05:13:39 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst worker1 # [ 87.100750] nix-grpc-daemon[860]: time=2026-09-23T05:13:39.607Z level=INFO msg="Upload complete. (326ms)" lb # [ 87.068711] niks3-server[967]: 2026/09/23 05:13:39 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete lb # [ 87.073282] niks3-server[967]: 2026/09/23 05:13:39 INFO Completed upload id=20 lb # [ 87.075190] niks3-server[967]: 2026/09/23 05:13:39 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo worker1 # [ 87.113406] nix-grpc-daemon[766]: ts=2026-09-23T05:13:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=7 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 worker1 # [ 87.136337] nix-grpc-daemon[766]: ts=2026-09-23T05:13:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=8 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 worker1 # [ 87.143437] nix-daemon[832]: reaped child process 1019, status = succeeded client # [ 87.220809] systemd[1]: share2.service: Deactivated successfully. client # [ 87.225809] systemd[1]: share2.service: Consumed 184ms CPU time over 7.553s wall clock time, 35.3M memory peak, 4K written to disk, 6.1K incoming IP traffic, 5.7K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share2, in 1.17 seconds) client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; } client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.06 seconds) (finished: subtest: the first client leaving does not fail the second, in 9.39 seconds) subtest: build hook: nix-daemon with builders = grpc://lb client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log client # [ 88.057706] systemd[1]: Started Nix Daemon. client # [ 88.212173] nix-daemon[1165]: accepted connection from pid 1162, user root (trusted) worker1 # [ 88.391587] nix-grpc-daemon[766]: ts=2026-09-23T05:13:40Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 88.399200] nix-grpc-daemon[766]: ts=2026-09-23T05:13:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 worker2 # [ 88.489726] nix-grpc-daemon[769]: ts=2026-09-23T05:13:40Z level=debug event=want client=11 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=0 worker2 # [ 88.490327] nix-grpc-daemon[769]: ts=2026-09-23T05:13:40Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 88.503920] nix-grpc-daemon[769]: ts=2026-09-23T05:13:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 88.525418] nix-grpc-daemon[769]: ts=2026-09-23T05:13:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 88.546311] nix-grpc-daemon[769]: ts=2026-09-23T05:13:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 88.566487] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 88.757645] nix-daemon[855]: accepted connection from pid 1161, user nix-grpc-daemon (trusted) worker2 # [ 88.784699] nix-daemon[855]: reaped child process 1164, status = succeeded lb # [ 88.682252] niks3-server[967]: 2026/09/23 05:13:41 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 88.797354] nix-grpc-daemon[870]: time=2026-09-23T05:13:41.247Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 88.799041] nix-grpc-daemon[870]: time=2026-09-23T05:13:41.247Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)" lb # [ 88.725664] niks3-server[967]: 2026/09/23 05:13:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign worker2 # [ 88.840389] nix-grpc-daemon[870]: time=2026-09-23T05:13:41.292Z level=INFO msg="Uploading 1 narinfos" lb # [ 88.744664] niks3-server[967]: 2026/09/23 05:13:41 INFO Signed narinfos id=21 count=1 lb # [ 88.750641] niks3-server[967]: 2026/09/23 05:13:41 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst lb # [ 88.769081] niks3-server[967]: 2026/09/23 05:13:41 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete worker2 # [ 88.880737] nix-grpc-daemon[870]: time=2026-09-23T05:13:41.333Z level=INFO msg="Upload complete. (292ms)" lb # [ 88.781969] niks3-server[967]: 2026/09/23 05:13:41 INFO Completed upload id=21 lb # [ 88.785284] niks3-server[967]: 2026/09/23 05:13:41 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls worker2 # [ 88.892436] nix-grpc-daemon[769]: ts=2026-09-23T05:13:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 88.792982] niks3-server[967]: 2026/09/23 05:13:41 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo worker2 # [ 88.904492] nix-grpc-daemon[769]: ts=2026-09-23T05:13:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 88.919995] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 88.932397] nix-daemon[855]: reaped child process 1159, status = succeeded worker2 # [ 88.939979] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) lb # [ 89.186809] envoy[895]: conn peer=[2001:db8:1::1]:48538 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2977 tx=812 ms=6987 lb # [ 89.192111] envoy[895]: conn peer=[2001:db8:1::1]:48544 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1443 tx=1197 ms=7328 worker2 # [ 90.643859] nix-daemon[855]: reaped child process 1168, status = succeeded worker2 # [ 90.736866] nix-daemon[855]: accepted connection from pid 1184, user nix-grpc-daemon (trusted) worker2 # [ 90.763248] nix-daemon[855]: reaped child process 1187, status = succeeded lb # [ 90.666302] niks3-server[967]: 2026/09/23 05:13:43 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 90.784000] nix-grpc-daemon[870]: time=2026-09-23T05:13:43.233Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 90.785897] nix-grpc-daemon[870]: time=2026-09-23T05:13:43.233Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)" lb # [ 90.725941] niks3-server[967]: 2026/09/23 05:13:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign worker2 # [ 90.839310] nix-grpc-daemon[870]: time=2026-09-23T05:13:43.291Z level=INFO msg="Uploading 1 narinfos" lb # [ 90.739076] niks3-server[967]: 2026/09/23 05:13:43 INFO Signed narinfos id=22 count=1 lb # [ 90.745183] niks3-server[967]: 2026/09/23 05:13:43 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv lb # [ 90.753544] niks3-server[967]: 2026/09/23 05:13:43 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls lb # [ 90.759365] niks3-server[967]: 2026/09/23 05:13:43 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst lb # [ 90.769315] niks3-server[967]: 2026/09/23 05:13:43 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete lb # [ 90.771819] niks3-server[967]: 2026/09/23 05:13:43 INFO Completed upload id=22 worker2 # [ 90.871435] nix-grpc-daemon[870]: time=2026-09-23T05:13:43.324Z level=INFO msg="Upload complete. (315ms)" lb # [ 90.776493] niks3-server[967]: 2026/09/23 05:13:43 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo worker2 # [ 90.892526] nix-grpc-daemon[769]: ts=2026-09-23T05:13:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=12 outputs=1 worker2 # [ 90.912653] nix-grpc-daemon[769]: ts=2026-09-23T05:13:43Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 90.918703] nix-daemon[855]: reaped child process 1166, status = succeeded worker1 # [ 90.883648] nix-grpc-daemon[766]: ts=2026-09-23T05:13:43Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 90.900184] nix-grpc-daemon[766]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'... worker1 # [ 90.923366] nix-grpc-daemon[766]: ts=2026-09-23T05:13:43Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=0 paths=1 nar_bytes_out=128 worker2 # [ 91.291207] nix-grpc-daemon[769]: ts=2026-09-23T05:13:43Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:58726 worker1 # [ 91.226087] nix-grpc-daemon[766]: ts=2026-09-23T05:13:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 91.239026] nix-grpc-daemon[860]: time=2026-09-23T05:13:43.744Z level=INFO msg="All 1 paths already cached" worker2 # [ 91.327699] nix-grpc-daemon[769]: ts=2026-09-23T05:13:43Z level=debug event=want client=12 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=0 worker2 # [ 91.334995] nix-grpc-daemon[769]: ts=2026-09-23T05:13:43Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 91.375353] nix-grpc-daemon[769]: ts=2026-09-23T05:13:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 91.392495] nix-grpc-daemon[769]: ts=2026-09-23T05:13:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 91.408235] nix-grpc-daemon[870]: time=2026-09-23T05:13:43.861Z level=INFO msg="All 1 paths already cached" worker2 # [ 91.418943] nix-grpc-daemon[769]: ts=2026-09-23T05:13:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 91.433570] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 91.601024] nix-daemon[855]: accepted connection from pid 1193, user nix-grpc-daemon (trusted) worker2 # [ 91.624345] nix-daemon[855]: reaped child process 1196, status = succeeded lb # [ 91.528686] niks3-server[967]: 2026/09/23 05:13:44 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 91.640094] nix-grpc-daemon[870]: time=2026-09-23T05:13:44.093Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 91.642128] nix-grpc-daemon[870]: time=2026-09-23T05:13:44.093Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)" lb # [ 91.571783] niks3-server[967]: 2026/09/23 05:13:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign worker2 # [ 91.681555] nix-grpc-daemon[870]: time=2026-09-23T05:13:44.134Z level=INFO msg="Uploading 1 narinfos" lb # [ 91.586485] niks3-server[967]: 2026/09/23 05:13:44 INFO Signed narinfos id=23 count=1 lb # [ 91.594356] niks3-server[967]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls lb # [ 91.608146] niks3-server[967]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst worker2 # [ 91.718449] nix-grpc-daemon[870]: time=2026-09-23T05:13:44.171Z level=INFO msg="Upload complete. (266ms)" lb # [ 91.617127] niks3-server[967]: 2026/09/23 05:13:44 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete lb # [ 91.626323] niks3-server[967]: 2026/09/23 05:13:44 INFO Completed upload id=23 lb # [ 91.629952] niks3-server[967]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo worker2 # [ 91.728458] nix-grpc-daemon[769]: ts=2026-09-23T05:13:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=0 paths=1 nar_bytes_in=624 worker2 # [ 91.749000] nix-grpc-daemon[769]: ts=2026-09-23T05:13:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 91.761006] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 91.765237] nix-daemon[855]: reaped child process 1191, status = succeeded worker2 # [ 91.780170] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 92.657409] nix-daemon[855]: reaped child process 1200, status = succeeded worker2 # [ 92.722143] nix-daemon[855]: accepted connection from pid 1216, user nix-grpc-daemon (trusted) worker2 # [ 92.745434] nix-daemon[855]: reaped child process 1219, status = succeeded lb # [ 92.650405] niks3-server[967]: 2026/09/23 05:13:45 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 92.764450] nix-grpc-daemon[870]: time=2026-09-23T05:13:45.215Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 92.765958] nix-grpc-daemon[870]: time=2026-09-23T05:13:45.216Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)" lb # [ 92.702791] niks3-server[967]: 2026/09/23 05:13:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign worker2 # [ 92.812947] nix-grpc-daemon[870]: time=2026-09-23T05:13:45.265Z level=INFO msg="Uploading 1 narinfos" lb # [ 92.715961] niks3-server[967]: 2026/09/23 05:13:45 INFO Signed narinfos id=24 count=1 lb # [ 92.721355] niks3-server[967]: 2026/09/23 05:13:45 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv lb # [ 92.730690] niks3-server[967]: 2026/09/23 05:13:45 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls lb # [ 92.740242] niks3-server[967]: 2026/09/23 05:13:45 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst worker2 # [ 92.850146] nix-grpc-daemon[870]: time=2026-09-23T05:13:45.303Z level=INFO msg="Upload complete. (259ms)" lb # [ 92.748771] niks3-server[967]: 2026/09/23 05:13:45 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete lb # [ 92.753409] niks3-server[967]: 2026/09/23 05:13:45 INFO Completed upload id=24 lb # [ 92.758088] niks3-server[967]: 2026/09/23 05:13:45 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo worker2 # [ 92.866272] nix-grpc-daemon[769]: ts=2026-09-23T05:13:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=13 outputs=1 worker1 # [ 92.802804] nix-grpc-daemon[766]: ts=2026-09-23T05:13:45Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49090 worker2 # [ 92.882805] nix-daemon[855]: reaped child process 1198, status = succeeded worker1 # [ 92.816781] nix-grpc-daemon[766]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'... worker1 # [ 92.858610] nix-grpc-daemon[766]: ts=2026-09-23T05:13:45Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 92.865886] nix-grpc-daemon[766]: ts=2026-09-23T05:13:45Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=0 paths=1 nar_bytes_out=176 worker2 # [ 93.220093] nix-grpc-daemon[769]: ts=2026-09-23T05:13:45Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:58726 worker1 # [ 93.155625] nix-grpc-daemon[766]: ts=2026-09-23T05:13:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 93.167105] nix-grpc-daemon[860]: time=2026-09-23T05:13:45.672Z level=INFO msg="All 2 paths already cached" worker2 # [ 93.255853] nix-grpc-daemon[769]: ts=2026-09-23T05:13:45Z level=debug event=want client=13 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=0 worker2 # [ 93.264348] nix-grpc-daemon[769]: ts=2026-09-23T05:13:45Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 93.271853] nix-grpc-daemon[769]: ts=2026-09-23T05:13:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 93.284097] nix-grpc-daemon[769]: ts=2026-09-23T05:13:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 93.298656] nix-grpc-daemon[870]: time=2026-09-23T05:13:45.751Z level=INFO msg="All 2 paths already cached" worker2 # [ 93.309959] nix-grpc-daemon[769]: ts=2026-09-23T05:13:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 93.324457] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 93.488886] nix-daemon[855]: accepted connection from pid 1225, user nix-grpc-daemon (trusted) worker2 # [ 93.515715] nix-daemon[855]: reaped child process 1228, status = succeeded lb # [ 93.411206] niks3-server[967]: 2026/09/23 05:13:45 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 93.523364] nix-grpc-daemon[870]: time=2026-09-23T05:13:45.976Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 93.525381] nix-grpc-daemon[870]: time=2026-09-23T05:13:45.976Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)" lb # [ 93.444314] niks3-server[967]: 2026/09/23 05:13:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign worker2 # [ 93.551988] nix-grpc-daemon[870]: time=2026-09-23T05:13:46.004Z level=INFO msg="Uploading 1 narinfos" lb # [ 93.458853] niks3-server[967]: 2026/09/23 05:13:45 INFO Signed narinfos id=25 count=1 lb # [ 93.466904] niks3-server[967]: 2026/09/23 05:13:45 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls lb # [ 93.478764] niks3-server[967]: 2026/09/23 05:13:45 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst worker2 # [ 93.588650] nix-grpc-daemon[870]: time=2026-09-23T05:13:46.041Z level=INFO msg="Upload complete. (246ms)" lb # [ 93.488723] niks3-server[967]: 2026/09/23 05:13:45 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete lb # [ 93.495025] niks3-server[967]: 2026/09/23 05:13:45 INFO Completed upload id=25 worker2 # [ 93.596412] nix-grpc-daemon[769]: ts=2026-09-23T05:13:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 93.498358] niks3-server[967]: 2026/09/23 05:13:45 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo worker2 # [ 93.608614] nix-grpc-daemon[769]: ts=2026-09-23T05:13:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 93.618865] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 93.626435] nix-daemon[855]: reaped child process 1223, status = succeeded worker2 # [ 93.639337] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 94.881843] nix-daemon[855]: reaped child process 1232, status = succeeded worker2 # [ 95.000144] nix-daemon[855]: accepted connection from pid 1248, user nix-grpc-daemon (trusted) worker2 # [ 95.024219] nix-daemon[855]: reaped child process 1251, status = succeeded lb # [ 94.921695] niks3-server[967]: 2026/09/23 05:13:47 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 95.034931] nix-grpc-daemon[870]: time=2026-09-23T05:13:47.486Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 95.036589] nix-grpc-daemon[870]: time=2026-09-23T05:13:47.486Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)" lb # [ 94.963742] niks3-server[967]: 2026/09/23 05:13:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign worker2 # [ 95.073430] nix-grpc-daemon[870]: time=2026-09-23T05:13:47.526Z level=INFO msg="Uploading 1 narinfos" lb # [ 94.978040] niks3-server[967]: 2026/09/23 05:13:47 INFO Signed narinfos id=26 count=1 lb # [ 94.984835] niks3-server[967]: 2026/09/23 05:13:47 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv lb # [ 94.994082] niks3-server[967]: 2026/09/23 05:13:47 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls lb # [ 95.003923] niks3-server[967]: 2026/09/23 05:13:47 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst worker2 # [ 95.111094] nix-grpc-daemon[870]: time=2026-09-23T05:13:47.564Z level=INFO msg="Upload complete. (257ms)" lb # [ 95.012170] niks3-server[967]: 2026/09/23 05:13:47 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete lb # [ 95.015569] niks3-server[967]: 2026/09/23 05:13:47 INFO Completed upload id=26 lb # [ 95.019717] niks3-server[967]: 2026/09/23 05:13:47 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo worker2 # [ 95.123289] nix-grpc-daemon[769]: ts=2026-09-23T05:13:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=14 outputs=1 worker1 # [ 95.062500] nix-grpc-daemon[766]: ts=2026-09-23T05:13:47Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49090 worker2 # [ 95.143688] nix-daemon[855]: reaped child process 1230, status = succeeded worker1 # [ 95.075770] nix-grpc-daemon[766]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'... worker1 # [ 95.117266] nix-grpc-daemon[766]: ts=2026-09-23T05:13:47Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 95.124293] nix-grpc-daemon[766]: ts=2026-09-23T05:13:47Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=0 paths=1 nar_bytes_out=232 client # [ 95.279222] nix-daemon[1165]: reaped child process 1167, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.41 seconds) client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.04 seconds) client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds) client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds) (finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.53 seconds) subtest: an input only one worker has still reaches the builder client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only client # this derivation will be built: client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.33 seconds) client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.13 seconds) worker1: must succeed: nix-store --import < /tmp/shared/inp.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.20 seconds) worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.03 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2 client # [ 96.311575] nix-daemon[1165]: accepted connection from pid 1248, user root (trusted) client # this derivation will be built: client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker2 # [ 96.523557] nix-grpc-daemon[769]: ts=2026-09-23T05:13:48Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:58726 client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 96.457944] nix-grpc-daemon[766]: ts=2026-09-23T05:13:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 96.633462] nix-daemon[832]: accepted connection from pid 1068, user nix-grpc-daemon (trusted) worker1 # [ 96.657376] nix-daemon[832]: reaped child process 1071, status = succeeded lb # [ 96.626930] niks3-server[967]: 2026/09/23 05:13:49 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 96.672566] nix-grpc-daemon[860]: time=2026-09-23T05:13:49.177Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 96.674582] nix-grpc-daemon[860]: time=2026-09-23T05:13:49.178Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)" lb # [ 96.667628] niks3-server[967]: 2026/09/23 05:13:49 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst worker1 # [ 96.714817] nix-grpc-daemon[860]: time=2026-09-23T05:13:49.220Z level=INFO msg="Uploading 1 narinfos" lb # [ 96.684397] niks3-server[967]: 2026/09/23 05:13:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign lb # [ 96.694797] niks3-server[967]: 2026/09/23 05:13:49 INFO Signed narinfos id=27 count=1 lb # [ 96.706242] niks3-server[967]: 2026/09/23 05:13:49 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls worker1 # [ 96.747694] nix-grpc-daemon[860]: time=2026-09-23T05:13:49.254Z level=INFO msg="Upload complete. (284ms)" lb # [ 96.718195] niks3-server[967]: 2026/09/23 05:13:49 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete lb # [ 96.725702] niks3-server[967]: 2026/09/23 05:13:49 INFO Completed upload id=27 lb # [ 96.729270] niks3-server[967]: 2026/09/23 05:13:49 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo worker2 # [ 96.844200] nix-grpc-daemon[769]: ts=2026-09-23T05:13:49Z level=debug event=want client=14 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=0 worker2 # [ 96.851306] nix-grpc-daemon[769]: ts=2026-09-23T05:13:49Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 96.831534] nix-grpc-daemon[766]: ts=2026-09-23T05:13:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 96.848626] nix-grpc-daemon[766]: ts=2026-09-23T05:13:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 client # copying 2 paths to 'lb:50051'... client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'... worker1 # [ 96.869275] nix-grpc-daemon[766]: ts=2026-09-23T05:13:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'... worker1 # [ 96.883286] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 97.027214] nix-daemon[832]: accepted connection from pid 1075, user nix-grpc-daemon (trusted) lb # [ 97.018615] niks3-server[967]: 2026/09/23 05:13:49 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 97.054265] nix-daemon[832]: reaped child process 1078, status = succeeded lb # [ 97.029272] niks3-server[967]: 2026/09/23 05:13:49 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 97.072122] nix-grpc-daemon[860]: time=2026-09-23T05:13:49.577Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker1 # [ 97.073827] nix-grpc-daemon[860]: time=2026-09-23T05:13:49.577Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)" worker1 # [ 97.075558] nix-grpc-daemon[860]: time=2026-09-23T05:13:49.577Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)" lb # [ 97.064056] niks3-server[967]: 2026/09/23 05:13:49 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst lb # [ 97.097296] niks3-server[967]: 2026/09/23 05:13:49 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls lb # [ 97.105854] niks3-server[967]: 2026/09/23 05:13:49 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst lb # [ 97.117295] niks3-server[967]: 2026/09/23 05:13:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign worker1 # [ 97.155498] nix-grpc-daemon[860]: time=2026-09-23T05:13:49.661Z level=INFO msg="Uploading 3 narinfos" lb # [ 97.125598] niks3-server[967]: 2026/09/23 05:13:49 INFO Signed narinfos id=28 count=1 lb # [ 97.130186] niks3-server[967]: 2026/09/23 05:13:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign lb # [ 97.137578] niks3-server[967]: 2026/09/23 05:13:49 INFO Signed narinfos id=29 count=2 lb # [ 97.141533] niks3-server[967]: 2026/09/23 05:13:49 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls lb # [ 97.156703] niks3-server[967]: 2026/09/23 05:13:49 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo lb # [ 97.164446] niks3-server[967]: 2026/09/23 05:13:49 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo lb # [ 97.170578] niks3-server[967]: 2026/09/23 05:13:49 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete lb # [ 97.174582] niks3-server[967]: 2026/09/23 05:13:49 INFO Completed upload id=28 lb # [ 97.179717] niks3-server[967]: 2026/09/23 05:13:49 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete lb # [ 97.183297] niks3-server[967]: 2026/09/23 05:13:49 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo worker1 # [ 97.214885] nix-grpc-daemon[860]: time=2026-09-23T05:13:49.721Z level=INFO msg="Upload complete. (311ms)" lb # [ 97.185961] niks3-server[967]: 2026/09/23 05:13:49 INFO Completed upload id=29 worker1 # [ 97.226039] nix-grpc-daemon[766]: ts=2026-09-23T05:13:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=0 paths=2 nar_bytes_in=984 worker1 # [ 97.239665] nix-grpc-daemon[766]: ts=2026-09-23T05:13:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 97.256812] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 97.266307] nix-daemon[832]: reaped child process 1073, status = succeeded client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051... client # uses-input-w1onlya> node-a: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker1 # [ 97.285755] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 98.603964] nix-daemon[832]: reaped child process 1082, status = succeeded worker1 # [ 98.677697] nix-daemon[832]: accepted connection from pid 1098, user nix-grpc-daemon (trusted) worker1 # [ 98.699929] nix-daemon[832]: reaped child process 1101, status = succeeded lb # [ 98.673992] niks3-server[967]: 2026/09/23 05:13:51 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 98.719314] nix-grpc-daemon[860]: time=2026-09-23T05:13:51.223Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 98.707501] niks3-server[967]: 2026/09/23 05:13:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign worker1 # [ 98.752744] nix-grpc-daemon[860]: time=2026-09-23T05:13:51.259Z level=INFO msg="Uploading 1 narinfos" lb # [ 98.725090] niks3-server[967]: 2026/09/23 05:13:51 INFO Signed narinfos id=30 count=1 lb # [ 98.735511] niks3-server[967]: 2026/09/23 05:13:51 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv lb # [ 98.746520] niks3-server[967]: 2026/09/23 05:13:51 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls worker1 # [ 98.791228] nix-grpc-daemon[860]: time=2026-09-23T05:13:51.297Z level=INFO msg="Upload complete. (291ms)" lb # [ 98.761132] niks3-server[967]: 2026/09/23 05:13:51 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete lb # [ 98.769669] niks3-server[967]: 2026/09/23 05:13:51 INFO Completed upload id=30 worker1 # [ 98.801133] nix-grpc-daemon[766]: ts=2026-09-23T05:13:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=15 outputs=1 client # copying 1 paths... lb # [ 98.774144] niks3-server[967]: 2026/09/23 05:13:51 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo worker1 # [ 98.819839] nix-grpc-daemon[766]: ts=2026-09-23T05:13:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 98.825390] nix-daemon[832]: reaped child process 1080, status = succeeded client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'... worker1 # [ 98.851928] nix-grpc-daemon[766]: ts=2026-09-23T05:13:51Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 98.854971] nix-grpc-daemon[766]: ts=2026-09-23T05:13:51Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=0 paths=1 nar_bytes_out=120 client # [ 99.009192] nix-daemon[1165]: reaped child process 1251, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.91 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2 client # [ 99.192593] nix-daemon[1165]: accepted connection from pid 1270, user root (trusted) client # this derivation will be built: client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv lb # [ 99.186138] envoy[895]: conn peer=[2001:db8:1::1]:47220 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=334 ms=109 lb # [ 99.191201] envoy[895]: conn peer=[2001:db8:1::1]:46398 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3427 tx=1968 ms=2598 lb # [ 99.196505] envoy[895]: conn peer=[2001:db8:1::1]:47228 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3887 tx=2158 ms=1705 lb # [ 99.202173] envoy[895]: conn peer=[2001:db8:1::1]:47236 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=375 ms=73 lb # [ 99.207510] envoy[895]: conn peer=[2001:db8:1::1]:47246 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4352 tx=2392 ms=2034 lb # [ 99.212984] envoy[895]: conn peer=[2001:db8:1::1]:47260 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=406 ms=69 lb # [ 99.218111] envoy[895]: conn peer=[2001:db8:1::1]:47268 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=65 lb # [ 99.224303] envoy[895]: conn peer=[2001:db8:1::1]:47266 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3862 tx=2065 ms=2462 worker1 # [ 99.338046] nix-grpc-daemon[766]: ts=2026-09-23T05:13:51Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:49090 client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker2 # [ 99.415910] nix-grpc-daemon[769]: ts=2026-09-23T05:13:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 99.445855] nix-grpc-daemon[769]: ts=2026-09-23T05:13:51Z level=debug event=want client=15 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=0 worker2 # [ 99.454601] nix-grpc-daemon[769]: ts=2026-09-23T05:13:51Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 99.466735] nix-grpc-daemon[769]: ts=2026-09-23T05:13:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 99.481263] nix-grpc-daemon[769]: ts=2026-09-23T05:13:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58726 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'... worker2 # [ 99.511490] nix-grpc-daemon[769]: ts=2026-09-23T05:13:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 99.530128] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 99.554292] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 38 ms (attempt 1/5) worker2 # [ 99.599934] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 100 ms (attempt 2/5) worker2 # [ 99.702220] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 153 ms (attempt 3/5) worker2 # [ 99.865015] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 56 ms (attempt 4/5) worker2 # [ 99.923505] nix-grpc-daemon[769]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 99.931795] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 99.938959] nix-grpc-daemon[769]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'... worker2 # [ 100.108942] nix-daemon[855]: accepted connection from pid 1267, user nix-grpc-daemon (trusted) worker2 # [ 100.134153] nix-daemon[855]: reaped child process 1270, status = succeeded lb # [ 100.039358] niks3-server[967]: 2026/09/23 05:13:52 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 100.153471] nix-grpc-daemon[870]: time=2026-09-23T05:13:52.605Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 100.155626] nix-grpc-daemon[870]: time=2026-09-23T05:13:52.605Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)" lb # [ 100.074119] niks3-server[967]: 2026/09/23 05:13:52 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst worker2 # [ 100.187671] nix-grpc-daemon[870]: time=2026-09-23T05:13:52.640Z level=INFO msg="Uploading 1 narinfos" lb # [ 100.090207] niks3-server[967]: 2026/09/23 05:13:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign lb # [ 100.100518] niks3-server[967]: 2026/09/23 05:13:52 INFO Signed narinfos id=31 count=1 lb # [ 100.106283] niks3-server[967]: 2026/09/23 05:13:52 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls worker2 # [ 100.215899] nix-grpc-daemon[870]: time=2026-09-23T05:13:52.669Z level=INFO msg="Upload complete. (257ms)" lb # [ 100.115978] niks3-server[967]: 2026/09/23 05:13:52 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete lb # [ 100.121164] niks3-server[967]: 2026/09/23 05:13:52 INFO Completed upload id=31 lb # [ 100.126931] niks3-server[967]: 2026/09/23 05:13:52 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo worker2 # [ 100.228429] nix-grpc-daemon[769]: ts=2026-09-23T05:13:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=0 paths=1 nar_bytes_in=568 worker2 # [ 100.238978] nix-grpc-daemon[769]: ts=2026-09-23T05:13:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 100.250922] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 100.257225] nix-daemon[855]: reaped child process 1260, status = succeeded worker2 # [ 100.265621] nix-grpc-daemon[769]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'... worker2 # [ 100.280076] nix-daemon[855]: accepted connection from pid 769, user nix-grpc-daemon (trusted) client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051... client # uses-input-w1onlyb> worker2: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 101.176374] nix-daemon[855]: reaped child process 1275, status = succeeded worker2 # [ 101.267264] nix-daemon[855]: accepted connection from pid 1291, user nix-grpc-daemon (trusted) worker2 # [ 101.290206] nix-daemon[855]: reaped child process 1294, status = succeeded lb # [ 101.186661] niks3-server[967]: 2026/09/23 05:13:53 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 101.299630] nix-grpc-daemon[870]: time=2026-09-23T05:13:53.751Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 101.219810] niks3-server[967]: 2026/09/23 05:13:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign worker2 # [ 101.333253] nix-grpc-daemon[870]: time=2026-09-23T05:13:53.786Z level=INFO msg="Uploading 1 narinfos" lb # [ 101.235881] niks3-server[967]: 2026/09/23 05:13:53 INFO Signed narinfos id=32 count=1 lb # [ 101.243288] niks3-server[967]: 2026/09/23 05:13:53 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv lb # [ 101.254462] niks3-server[967]: 2026/09/23 05:13:53 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls worker2 # [ 101.360610] nix-grpc-daemon[870]: time=2026-09-23T05:13:53.813Z level=INFO msg="Upload complete. (267ms)" lb # [ 101.261595] niks3-server[967]: 2026/09/23 05:13:53 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete lb # [ 101.267882] niks3-server[967]: 2026/09/23 05:13:53 INFO Completed upload id=32 lb # [ 101.270662] niks3-server[967]: 2026/09/23 05:13:53 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo client # copying 1 paths... worker2 # [ 101.373572] nix-grpc-daemon[769]: ts=2026-09-23T05:13:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=16 outputs=1 worker1 # [ 101.313535] nix-grpc-daemon[766]: ts=2026-09-23T05:13:53Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49090 worker2 # [ 101.388420] nix-daemon[855]: reaped child process 1272, status = succeeded worker1 # [ 101.326748] nix-grpc-daemon[766]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'... client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'... worker1 # [ 101.371119] nix-grpc-daemon[766]: ts=2026-09-23T05:13:53Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 101.378429] nix-grpc-daemon[766]: ts=2026-09-23T05:13:53Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=0 paths=1 nar_bytes_out=120 client # [ 101.532364] nix-daemon[1165]: reaped child process 1273, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 2.52 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2 client # [ 101.715399] nix-daemon[1165]: accepted connection from pid 1292, user root (trusted) client # this derivation will be built: client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker2 # [ 101.921861] nix-grpc-daemon[769]: ts=2026-09-23T05:13:54Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:58726 client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 101.856910] nix-grpc-daemon[766]: ts=2026-09-23T05:13:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 101.867945] nix-grpc-daemon[860]: time=2026-09-23T05:13:54.373Z level=INFO msg="All 1 paths already cached" worker2 # [ 101.952158] nix-grpc-daemon[769]: ts=2026-09-23T05:13:54Z level=debug event=want client=16 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=0 worker2 # [ 101.958993] nix-grpc-daemon[769]: ts=2026-09-23T05:13:54Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 101.892808] nix-grpc-daemon[766]: ts=2026-09-23T05:13:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 101.909017] nix-grpc-daemon[766]: ts=2026-09-23T05:13:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 101.924092] nix-grpc-daemon[860]: time=2026-09-23T05:13:54.430Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'... worker1 # [ 101.935520] nix-grpc-daemon[766]: ts=2026-09-23T05:13:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 101.951667] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 102.115609] nix-daemon[832]: accepted connection from pid 1106, user nix-grpc-daemon (trusted) worker1 # [ 102.134800] nix-daemon[832]: reaped child process 1109, status = succeeded lb # [ 102.110288] niks3-server[967]: 2026/09/23 05:13:54 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 102.151877] nix-grpc-daemon[860]: time=2026-09-23T05:13:54.658Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 102.153681] nix-grpc-daemon[860]: time=2026-09-23T05:13:54.658Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)" worker1 # [ 102.173526] nix-grpc-daemon[860]: time=2026-09-23T05:13:54.679Z level=INFO msg="Uploading 1 narinfos" lb # [ 102.141205] niks3-server[967]: 2026/09/23 05:13:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign lb # [ 102.156134] niks3-server[967]: 2026/09/23 05:13:54 INFO Signed narinfos id=33 count=1 lb # [ 102.164200] niks3-server[967]: 2026/09/23 05:13:54 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls lb # [ 102.172898] niks3-server[967]: 2026/09/23 05:13:54 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst lb # [ 102.183323] niks3-server[967]: 2026/09/23 05:13:54 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete worker1 # [ 102.220823] nix-grpc-daemon[860]: time=2026-09-23T05:13:54.727Z level=INFO msg="Upload complete. (250ms)" lb # [ 102.190568] niks3-server[967]: 2026/09/23 05:13:54 INFO Completed upload id=33 lb # [ 102.198460] niks3-server[967]: 2026/09/23 05:13:54 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo worker1 # [ 102.231357] nix-grpc-daemon[766]: ts=2026-09-23T05:13:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=0 paths=1 nar_bytes_in=568 worker1 # [ 102.240308] nix-grpc-daemon[766]: ts=2026-09-23T05:13:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 worker1 # [ 102.253004] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 102.258235] nix-daemon[832]: reaped child process 1104, status = succeeded client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051... worker1 # [ 102.273266] nix-daemon[832]: accepted connection from pid 766, user nix-grpc-daemon (trusted) client # uses-input-w1onlyc> node-a: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 103.141428] nix-daemon[832]: reaped child process 1113, status = succeeded worker1 # [ 103.262375] nix-daemon[832]: accepted connection from pid 1129, user nix-grpc-daemon (trusted) worker1 # [ 103.281974] nix-daemon[832]: reaped child process 1132, status = succeeded lb # [ 103.261774] niks3-server[967]: 2026/09/23 05:13:55 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 103.308085] nix-grpc-daemon[860]: time=2026-09-23T05:13:55.812Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" worker1 # [ 103.332600] nix-grpc-daemon[860]: time=2026-09-23T05:13:55.838Z level=INFO msg="Uploading 1 narinfos" lb # [ 103.300102] niks3-server[967]: 2026/09/23 05:13:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign lb # [ 103.312447] niks3-server[967]: 2026/09/23 05:13:55 INFO Signed narinfos id=34 count=1 lb # [ 103.315259] niks3-server[967]: 2026/09/23 05:13:55 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls lb # [ 103.329942] niks3-server[967]: 2026/09/23 05:13:55 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker1 # [ 103.367926] nix-grpc-daemon[860]: time=2026-09-23T05:13:55.874Z level=INFO msg="Upload complete. (283ms)" lb # [ 103.338363] niks3-server[967]: 2026/09/23 05:13:55 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete lb # [ 103.347053] niks3-server[967]: 2026/09/23 05:13:55 INFO Completed upload id=34 client # copying 1 paths... lb # [ 103.349321] niks3-server[967]: 2026/09/23 05:13:55 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo worker1 # [ 103.380969] nix-grpc-daemon[766]: ts=2026-09-23T05:13:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49090 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=17 outputs=1 worker2 # [ 103.457518] nix-grpc-daemon[769]: ts=2026-09-23T05:13:55Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:58726 worker1 # [ 103.400956] nix-daemon[832]: reaped child process 1111, status = succeeded worker2 # [ 103.470650] nix-grpc-daemon[769]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'... client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'... worker2 # [ 103.511478] nix-grpc-daemon[769]: ts=2026-09-23T05:13:55Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:58726 worker2 # [ 103.518888] nix-grpc-daemon[769]: ts=2026-09-23T05:13:55Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:58726 duration_s=0 paths=1 nar_bytes_out=120 client # [ 103.601852] nix-daemon[1165]: reaped child process 1295, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.07 seconds) (finished: subtest: an input only one worker has still reaches the builder, in 8.19 seconds) subtest: upload whose reference the worker lacks is completed from the cache client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache client # this derivation will be built: client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.36 seconds) client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache client # this derivation will be built: client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.31 seconds) client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.13 seconds) worker1: must succeed: nix-store --import < /tmp/shared/ref.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.20 seconds) worker2 # [ 104.727056] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 105.671313] nix-grpc-daemon[769]: ts=2026-09-23T05:13:58Z level=info event=signal_exit lb # [ 105.950449] niks3-server[967]: 2026/09/23 05:13:58 INFO lead: released remote=192.168.1.4:38300 worker1 # [ 106.979460] nix-grpc-daemon[766]: ts=2026-09-23T05:13:59Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 107.056653] nix-grpc-daemon[769]: ts=2026-09-23T05:13:59Z level=info event=scheduler_restarting worker2 # [ 107.057102] nix-grpc-daemon[769]: ts=2026-09-23T05:13:59Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:58724 worker2 # [ 107.065233] nix-grpc-daemon[769]: ts=2026-09-23T05:13:59Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 107.065460] nix-grpc-daemon[769]: ts=2026-09-23T05:13:59Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 107.065592] nix-grpc-daemon[769]: ts=2026-09-23T05:13:59Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:58724 worker2 # [ 107.065703] nix-grpc-daemon[769]: ts=2026-09-23T05:13:59Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 107.093055] nix-daemon[855]: reaped child process 857, status = succeeded worker1 # [ 107.185201] nix-grpc-daemon[766]: ts=2026-09-23T05:13:59Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 107.190711] nix-grpc-daemon[766]: ts=2026-09-23T05:13:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 107.278380] nix-grpc-daemon[769]: ts=2026-09-23T05:13:59Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 107.426591] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 107.433244] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 107.435849] systemd[1]: nix-grpc-daemon.service: Consumed 3.889s CPU time over 1min 27.994s wall clock time, 70.1M memory peak, 112.5K incoming IP traffic, 273.1K outgoing IP traffic. worker2 # [ 107.454652] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 107.464297] systemd[1]: Closed Nix worker-protocol over gRPC. worker1 # [ 107.391586] nix-grpc-daemon[766]: ts=2026-09-23T05:13:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 107.465778] systemd[1]: nix-grpc-daemon.socket: Consumed 623us CPU time over 1min 36.996s wall clock time, 256K memory peak, 120.6K incoming IP traffic, 99.8K outgoing IP traffic. worker1 # [ 107.398083] nix-grpc-daemon[766]: ts=2026-09-23T05:13:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds) client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2 worker1 # [ 107.598465] nix-grpc-daemon[766]: ts=2026-09-23T05:14:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 107.604433] nix-grpc-daemon[766]: ts=2026-09-23T05:14:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 107.686801] nix-grpc-daemon[766]: ts=2026-09-23T05:14:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:49090 client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache worker1 # [ 107.694456] nix-grpc-daemon[766]: ts=2026-09-23T05:14:00Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49090 client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.25 seconds) worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds) worker1 # [ 107.805035] nix-grpc-daemon[766]: ts=2026-09-23T05:14:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 107.811508] nix-grpc-daemon[766]: ts=2026-09-23T05:14:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 107.886168] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 107.888886] systemd[1]: Stopping Nix worker-protocol over gRPC... worker1 # [ 108.014397] nix-grpc-daemon[766]: ts=2026-09-23T05:14:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 108.019328] nix-grpc-daemon[766]: ts=2026-09-23T05:14:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 108.220312] nix-grpc-daemon[766]: ts=2026-09-23T05:14:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 108.225757] nix-grpc-daemon[766]: ts=2026-09-23T05:14:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 108.426618] nix-grpc-daemon[766]: ts=2026-09-23T05:14:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 108.432223] nix-grpc-daemon[766]: ts=2026-09-23T05:14:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 108.455469] nix-grpc-daemon[766]: ts=2026-09-23T05:14:00Z level=info event=signal_exit worker1 # [ 108.632893] nix-grpc-daemon[766]: ts=2026-09-23T05:14:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 108.638353] nix-grpc-daemon[766]: ts=2026-09-23T05:14:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 108.847249] systemd[1]: Starting scheduler-token.service... worker1 # [ 108.839102] nix-grpc-daemon[766]: ts=2026-09-23T05:14:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 108.844601] nix-grpc-daemon[766]: ts=2026-09-23T05:14:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 108.932936] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 108.940228] systemd[1]: Finished scheduler-token.service. worker2 # [ 108.942285] systemd[1]: scheduler-token.service: Consumed 58ms CPU time over 90ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 108.965900] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 108.979573] nix-grpc-daemon[766]: ts=2026-09-23T05:14:01Z level=info event=scheduler_take_over lb # [ 108.956758] niks3-server[967]: 2026/09/23 05:14:01 INFO lead: acquired remote=192.168.1.3:35060 lb # [ 108.957226] niks3-server[967]: 2026/09/23 05:14:01 INFO lead: released remote=192.168.1.3:35060 worker2 # [ 109.084472] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:01Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 109.086764] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:01Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 109.090649] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:01Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 109.096058] nix-grpc-daemon[1321]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 109.097531] nix-grpc-daemon[1321]: E0000 00:00:1790140441.548669 1321 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 109.111980] nix-grpc-daemon[1321]: E0000 00:00:1790140441.554894 1321 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb # [ 109.011422] niks3-server[967]: 2026/09/23 05:14:01 INFO lead: acquired remote=192.168.1.4:35920 worker1 # [ 109.046615] nix-grpc-daemon[766]: ts=2026-09-23T05:14:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 109.052197] nix-grpc-daemon[766]: ts=2026-09-23T05:14:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 109.144193] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:01Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 109.145995] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 109.146806] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:01Z level=info event=scheduler_take_over worker2 # [ 109.147922] nix-grpc-daemon[1321]: E0000 00:00:1790140441.568461 1361 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 109.156822] nix-grpc-daemon[1321]: E0000 00:00:1790140441.587413 1361 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 109.159778] nix-grpc-daemon[1321]: E0000 00:00:1790140441.587588 1361 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 109.162548] nix-grpc-daemon[1321]: E0000 00:00:1790140441.608823 1361 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 109.170758] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 109.178720] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:01Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 109.180264] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:01Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:56666 worker2 # [ 109.181948] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:01Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb # [ 109.185227] envoy[895]: conn peer=[2001:db8:1::1]:47274 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3727 tx=2127 ms=2100 lb # [ 109.190407] envoy[895]: conn peer=[2001:db8:1::1]:39868 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=71 lb # [ 109.195275] envoy[895]: conn peer=[2001:db8:1::1]:39878 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3780 tx=2120 ms=1656 lb # [ 109.200485] envoy[895]: conn peer=[2001:db8:1::1]:39888 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=70 lb # [ 109.205136] envoy[895]: conn peer=[2001:db8:1::4]:55260 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=57813 tx=3562 ms=87562 lb # [ 109.210107] envoy[895]: conn peer=[2001:db8:1::1]:39898 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=22 worker1 # [ 109.253153] nix-grpc-daemon[766]: ts=2026-09-23T05:14:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 109.325351] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:01Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:56666 worker2 # [ 109.330747] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:01Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 109.981220] nix-grpc-daemon[766]: ts=2026-09-23T05:14:02Z level=info event=scheduler_restarting worker2 # [ 110.063557] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:02Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:56666 worker2 # [ 110.068567] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:02Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 110.021822] nix-grpc-daemon[766]: ts=2026-09-23T05:14:02Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker1 # [ 110.042681] nix-daemon[832]: reaped child process 834, status = succeeded worker1 # [ 110.623176] nix-grpc-daemon[766]: E0000 00:00:1790140443.128338 812 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true worker1 # [ 110.630914] nix-grpc-daemon[766]: E0000 00:00:1790140443.137442 812 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker1 # [ 111.297500] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 111.304754] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 111.307428] systemd[1]: nix-grpc-daemon.service: Consumed 3.011s CPU time over 1min 32.071s wall clock time, 61.3M memory peak, 90.1K incoming IP traffic, 158.1K outgoing IP traffic. worker1 # [ 111.324983] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker1 # [ 111.331522] systemd[1]: Closed Nix worker-protocol over gRPC. worker1 # [ 111.333868] systemd[1]: nix-grpc-daemon.socket: Consumed 669us CPU time over 1min 40.914s wall clock time, 256K memory peak, 98.6K incoming IP traffic, 88.6K outgoing IP traffic. lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.08 seconds) client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2 worker2 # [ 111.717359] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47682 worker2 # [ 111.733268] nix-daemon[855]: accepted connection from pid 1321, user nix-grpc-daemon (trusted) worker2 # [ 111.763844] nix-grpc-daemon[1321]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 99 ms (attempt 1/5) worker2 # [ 111.871319] nix-grpc-daemon[1321]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 153 ms (attempt 2/5) worker2 # [ 112.026604] nix-grpc-daemon[1321]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 329 ms (attempt 3/5) worker2 # [ 112.357914] nix-grpc-daemon[1321]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 782 ms (attempt 4/5) worker2 # [ 113.143342] nix-grpc-daemon[1321]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 113.148811] nix-grpc-daemon[1321]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org client # copying 2 paths... client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'... client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'... worker2 # [ 113.173517] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47682 worker2 # [ 113.190247] nix-daemon[855]: accepted connection from pid 1321, user nix-grpc-daemon (trusted) worker2 # [ 113.231664] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:05Z level=info event=niks3_push_started pid=1382 worker2 # [ 113.373121] nix-grpc-daemon[1382]: time=2026-09-23T05:14:05.825Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 113.484183] nix-daemon[855]: accepted connection from pid 1395, user nix-grpc-daemon (trusted) worker2 # [ 113.505342] nix-daemon[855]: reaped child process 1398, status = succeeded lb # [ 113.408730] niks3-server[967]: 2026/09/23 05:14:05 INFO Received uploads request method=POST path=/api/pending_closures lb # [ 113.420725] niks3-server[967]: 2026/09/23 05:14:05 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 113.534993] nix-grpc-daemon[1382]: time=2026-09-23T05:14:05.986Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker2 # [ 113.539295] nix-grpc-daemon[1382]: time=2026-09-23T05:14:05.986Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)" worker2 # [ 113.542208] nix-grpc-daemon[1382]: time=2026-09-23T05:14:05.986Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)" lb # [ 113.482333] niks3-server[967]: 2026/09/23 05:14:05 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst lb # [ 113.502297] niks3-server[967]: 2026/09/23 05:14:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign lb # [ 113.516611] niks3-server[967]: 2026/09/23 05:14:06 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls lb # [ 113.525663] niks3-server[967]: 2026/09/23 05:14:06 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst worker2 # [ 113.630828] nix-grpc-daemon[1382]: time=2026-09-23T05:14:06.080Z level=INFO msg="Uploading 3 narinfos" lb # [ 113.536104] niks3-server[967]: 2026/09/23 05:14:06 INFO Signed narinfos id=36 count=2 lb # [ 113.537513] niks3-server[967]: 2026/09/23 05:14:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign lb # [ 113.539368] niks3-server[967]: 2026/09/23 05:14:06 INFO Signed narinfos id=35 count=1 lb # [ 113.546314] niks3-server[967]: 2026/09/23 05:14:06 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls lb # [ 113.558508] niks3-server[967]: 2026/09/23 05:14:06 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete lb # [ 113.563993] niks3-server[967]: 2026/09/23 05:14:06 INFO Completed upload id=35 lb # [ 113.567347] niks3-server[967]: 2026/09/23 05:14:06 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete lb # [ 113.572697] niks3-server[967]: 2026/09/23 05:14:06 INFO Completed upload id=36 worker2 # [ 113.674932] nix-grpc-daemon[1382]: time=2026-09-23T05:14:06.127Z level=INFO msg="Upload complete. (298ms)" lb # [ 113.576692] niks3-server[967]: 2026/09/23 05:14:06 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo lb # [ 113.580262] niks3-server[967]: 2026/09/23 05:14:06 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo lb # [ 113.583624] niks3-server[967]: 2026/09/23 05:14:06 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo worker2 # [ 113.688410] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47682 duration_s=0 paths=2 nar_bytes_in=312 worker2 # [ 113.705951] nix-daemon[855]: reaped child process 1380, status = succeeded client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 2.24 seconds) worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds) worker1 # [ 113.767377] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 113.778508] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 113.931152] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:06Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 113.933859] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:06Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 113.938222] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:06Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 113.944071] nix-grpc-daemon[1155]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 113.945623] nix-grpc-daemon[1155]: E0000 00:00:1790140446.449519 1155 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 113.991505] nix-grpc-daemon[1155]: E0000 00:00:1790140446.456155 1155 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 113.995126] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:06Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 113.997530] nix-grpc-daemon[1155]: E0000 00:00:1790140446.463128 1195 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 114.000287] nix-grpc-daemon[1155]: E0000 00:00:1790140446.481295 1195 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 114.002522] nix-grpc-daemon[1155]: E0000 00:00:1790140446.481495 1195 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 114.004906] nix-grpc-daemon[1155]: E0000 00:00:1790140446.498054 1195 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 114.010567] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 114.013551] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:06Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 114.085937] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:06Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:56666 worker2 # [ 114.091606] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:06Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.15 seconds) (finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.54 seconds) subtest: a niks3 restart does not cost the leader its role lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb # [ 115.224853] systemd[1]: Stopping niks3 server... lb # [ 115.226739] niks3-server[967]: 2026/09/23 05:14:07 INFO Shutdown signal received, draining in-flight requests timeout=10s worker1 # [ 115.259965] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:07Z level=info event=scheduler_lock_lost status=200 lb # [ 115.234766] niks3-server[967]: 2026/09/23 05:14:07 INFO lead: released remote=192.168.1.4:35920 worker2 # [ 115.338148] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:07Z level=info event=scheduler_lock_lost status=200 lb # [ 115.252953] systemd[1]: niks3.service: Deactivated successfully. lb # [ 115.255718] systemd[1]: Stopped niks3 server. lb # [ 115.256764] systemd[1]: niks3.service: Consumed 714ms CPU time over 1min 31.557s wall clock time, 27.7M memory peak, 97.7K incoming IP traffic, 140.9K outgoing IP traffic. lb # [ 115.268150] systemd[1]: Starting niks3 server... lb # [ 115.331555] niks3-server[1111]: 2026/09/23 05:14:07 goose: no migrations to run. current version: 20260920000000 lb # [ 115.336767] niks3-server[1111]: 2026/09/23 05:14:07 OK 1_commit_pending_closure.sql (6.79ms) lb # [ 115.340505] niks3-server[1111]: 2026/09/23 05:14:07 OK 2_object_stats_trigger.sql (3.39ms) lb # [ 115.342782] niks3-server[1111]: 2026/09/23 05:14:07 goose: up to current file version: 2 lb # [ 115.343863] niks3-server[1111]: 2026/09/23 05:14:07 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key lb # [ 115.353786] niks3-server[1111]: 2026/09/23 05:14:07 INFO Read proxy enabled — serving cache objects from S3 lb # [ 115.355132] niks3-server[1111]: 2026/09/23 05:14:07 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 115.361298] niks3-server[1111]: 2026/09/23 05:14:07 INFO systemd watchdog enabled interval=15s lb # [ 115.364152] niks3-server[1111]: 2026/09/23 05:14:07 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 115.365668] systemd[1]: Started niks3 server. lb: waiting for unit niks3.service lb: (finished: waiting for unit niks3.service, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.07 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2 worker2 # [ 116.007212] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:47682 client # don't know how to build these paths: client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker1 # [ 115.956073] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 115.969782] nix-daemon[832]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) worker1 # [ 115.996632] nix-grpc-daemon[1155]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 74 ms (attempt 1/5) worker1 # [ 116.079108] nix-grpc-daemon[1155]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 198 ms (attempt 2/5) lb # [ 116.240554] niks3-server[1111]: 2026/09/23 05:14:08 INFO lead: acquired remote=192.168.1.4:40674 worker1 # [ 116.280342] nix-grpc-daemon[1155]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 140 ms (attempt 3/5) worker1 # [ 116.421284] nix-grpc-daemon[1155]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 16 ms (attempt 4/5) worker1 # [ 116.439431] nix-grpc-daemon[1155]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 116.446568] nix-grpc-daemon[1155]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 116.568893] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:09Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=0 worker2 # [ 116.574466] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:09Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 116.611069] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47682 worker2 # [ 116.629395] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47682 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'... worker2 # [ 116.645713] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47682 worker2 # [ 116.661094] nix-daemon[855]: accepted connection from pid 1321, user nix-grpc-daemon (trusted) worker2 # [ 116.827483] nix-daemon[855]: accepted connection from pid 1417, user nix-grpc-daemon (trusted) worker2 # [ 116.848085] nix-daemon[855]: reaped child process 1420, status = succeeded lb # [ 116.751125] niks3-server[1111]: 2026/09/23 05:14:09 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 116.867599] nix-grpc-daemon[1382]: time=2026-09-23T05:14:09.319Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 116.869491] nix-grpc-daemon[1382]: time=2026-09-23T05:14:09.319Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)" lb # [ 116.801921] niks3-server[1111]: 2026/09/23 05:14:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign lb # [ 116.817255] niks3-server[1111]: 2026/09/23 05:14:09 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls lb # [ 116.841888] niks3-server[1111]: 2026/09/23 05:14:09 INFO Signed narinfos id=37 count=1 worker2 # [ 116.947701] nix-grpc-daemon[1382]: time=2026-09-23T05:14:09.399Z level=INFO msg="Uploading 1 narinfos" lb # [ 116.852850] niks3-server[1111]: 2026/09/23 05:14:09 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst lb # [ 116.863435] niks3-server[1111]: 2026/09/23 05:14:09 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete lb # [ 116.868917] niks3-server[1111]: 2026/09/23 05:14:09 INFO Completed upload id=37 worker2 # [ 116.970270] nix-grpc-daemon[1382]: time=2026-09-23T05:14:09.423Z level=INFO msg="Upload complete. (292ms)" lb # [ 116.874153] niks3-server[1111]: 2026/09/23 05:14:09 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo worker2 # [ 116.980451] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47682 duration_s=0 paths=1 nar_bytes_in=544 worker2 # [ 117.001036] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47682 worker2 # [ 117.014449] nix-daemon[855]: accepted connection from pid 1321, user nix-grpc-daemon (trusted) worker2 # [ 117.018153] nix-daemon[855]: reaped child process 1415, status = succeeded client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051... worker2 # [ 117.031045] nix-daemon[855]: accepted connection from pid 1321, user nix-grpc-daemon (trusted) client # farm-a> worker2: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-after-niks3-restart worker2 # [ 118.079801] nix-daemon[855]: reaped child process 1424, status = succeeded worker2 # [ 118.192095] nix-daemon[855]: accepted connection from pid 1440, user nix-grpc-daemon (trusted) worker2 # [ 118.213351] nix-daemon[855]: reaped child process 1443, status = succeeded lb # [ 118.117504] niks3-server[1111]: 2026/09/23 05:14:10 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 118.235860] nix-grpc-daemon[1382]: time=2026-09-23T05:14:10.685Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 118.237828] nix-grpc-daemon[1382]: time=2026-09-23T05:14:10.685Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)" lb # [ 118.163420] niks3-server[1111]: 2026/09/23 05:14:10 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst worker2 # [ 118.281160] nix-grpc-daemon[1382]: time=2026-09-23T05:14:10.733Z level=INFO msg="Uploading 1 narinfos" lb # [ 118.179055] niks3-server[1111]: 2026/09/23 05:14:10 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls lb # [ 118.188297] niks3-server[1111]: 2026/09/23 05:14:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign lb # [ 118.191741] niks3-server[1111]: 2026/09/23 05:14:10 INFO Signed narinfos id=38 count=1 worker2 # [ 118.305952] nix-grpc-daemon[1382]: time=2026-09-23T05:14:10.759Z level=INFO msg="Upload complete. (261ms)" lb # [ 118.205073] niks3-server[1111]: 2026/09/23 05:14:10 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv lb # [ 118.208924] niks3-server[1111]: 2026/09/23 05:14:10 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete lb # [ 118.215989] niks3-server[1111]: 2026/09/23 05:14:10 INFO Completed upload id=38 lb # [ 118.218743] niks3-server[1111]: 2026/09/23 05:14:10 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo worker2 # [ 118.325772] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47682 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=1 worker2 # [ 118.343197] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:10Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=0 worker2 # [ 118.347716] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:10Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 118.352384] nix-daemon[855]: reaped child process 1422, status = succeeded worker2 # [ 118.382420] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47682 worker2 # [ 118.398288] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47682 worker2 # [ 118.414013] nix-grpc-daemon[1382]: time=2026-09-23T05:14:10.867Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'... worker2 # [ 118.425094] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47682 worker2 # [ 118.439222] nix-daemon[855]: accepted connection from pid 1321, user nix-grpc-daemon (trusted) worker2 # [ 118.594405] nix-daemon[855]: accepted connection from pid 1449, user nix-grpc-daemon (trusted) worker2 # [ 118.618124] nix-daemon[855]: reaped child process 1452, status = succeeded lb # [ 118.514439] niks3-server[1111]: 2026/09/23 05:14:11 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 118.628108] nix-grpc-daemon[1382]: time=2026-09-23T05:14:11.080Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 118.630029] nix-grpc-daemon[1382]: time=2026-09-23T05:14:11.080Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)" lb # [ 118.547170] niks3-server[1111]: 2026/09/23 05:14:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign worker2 # [ 118.657485] nix-grpc-daemon[1382]: time=2026-09-23T05:14:11.110Z level=INFO msg="Uploading 1 narinfos" lb # [ 118.565183] niks3-server[1111]: 2026/09/23 05:14:11 INFO Signed narinfos id=39 count=1 lb # [ 118.573612] niks3-server[1111]: 2026/09/23 05:14:11 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls lb # [ 118.583409] niks3-server[1111]: 2026/09/23 05:14:11 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst lb # [ 118.595996] niks3-server[1111]: 2026/09/23 05:14:11 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete lb # [ 118.600572] niks3-server[1111]: 2026/09/23 05:14:11 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo worker2 # [ 118.702453] nix-grpc-daemon[1382]: time=2026-09-23T05:14:11.155Z level=INFO msg="Upload complete. (245ms)" lb # [ 118.604080] niks3-server[1111]: 2026/09/23 05:14:11 INFO Completed upload id=39 worker2 # [ 118.713352] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47682 duration_s=0 paths=1 nar_bytes_in=656 worker2 # [ 118.732298] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47682 worker2 # [ 118.741283] nix-daemon[855]: accepted connection from pid 1321, user nix-grpc-daemon (trusted) worker2 # [ 118.746456] nix-daemon[855]: reaped child process 1447, status = succeeded client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051... worker2 # [ 118.757274] nix-daemon[855]: accepted connection from pid 1321, user nix-grpc-daemon (trusted) client # farm-b> worker2: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv lb # [ 119.187682] envoy[895]: conn peer=[2001:db8:1::3]:50050 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=13482 tx=3746 ms=90521 lb # [ 119.193713] envoy[895]: conn peer=[2001:db8:1::1]:60900 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1006 tx=282 ms=1988 client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-after-niks3-restart worker2 # [ 120.212221] nix-daemon[855]: reaped child process 1456, status = succeeded worker2 # [ 120.318477] nix-daemon[855]: accepted connection from pid 1472, user nix-grpc-daemon (trusted) worker2 # [ 120.338683] nix-daemon[855]: reaped child process 1475, status = succeeded lb # [ 120.244728] niks3-server[1111]: 2026/09/23 05:14:12 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 120.356600] nix-grpc-daemon[1382]: time=2026-09-23T05:14:12.809Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 120.358218] nix-grpc-daemon[1382]: time=2026-09-23T05:14:12.809Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)" lb # [ 120.291434] niks3-server[1111]: 2026/09/23 05:14:12 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls worker2 # [ 120.404701] nix-grpc-daemon[1382]: time=2026-09-23T05:14:12.857Z level=INFO msg="Uploading 1 narinfos" lb # [ 120.304319] niks3-server[1111]: 2026/09/23 05:14:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign lb # [ 120.316319] niks3-server[1111]: 2026/09/23 05:14:12 INFO Signed narinfos id=40 count=1 lb # [ 120.318561] niks3-server[1111]: 2026/09/23 05:14:12 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv lb # [ 120.328659] niks3-server[1111]: 2026/09/23 05:14:12 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst lb # [ 120.336485] niks3-server[1111]: 2026/09/23 05:14:12 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete lb # [ 120.341168] niks3-server[1111]: 2026/09/23 05:14:12 INFO Completed upload id=40 worker2 # [ 120.443246] nix-grpc-daemon[1382]: time=2026-09-23T05:14:12.896Z level=INFO msg="Upload complete. (272ms)" lb # [ 120.344185] niks3-server[1111]: 2026/09/23 05:14:12 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo worker2 # [ 120.459306] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47682 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=1 worker2 # [ 120.483506] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:12Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=0 worker2 # [ 120.488681] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:12Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 120.494033] nix-daemon[855]: reaped child process 1454, status = succeeded worker2 # [ 120.518654] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47682 worker2 # [ 120.534616] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47682 worker2 # [ 120.550263] nix-grpc-daemon[1382]: time=2026-09-23T05:14:13.003Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'... worker2 # [ 120.561830] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47682 worker2 # [ 120.576288] nix-daemon[855]: accepted connection from pid 1321, user nix-grpc-daemon (trusted) worker2 # [ 120.730321] nix-daemon[855]: accepted connection from pid 1481, user nix-grpc-daemon (trusted) worker2 # [ 120.754278] nix-daemon[855]: reaped child process 1484, status = succeeded lb # [ 120.650696] niks3-server[1111]: 2026/09/23 05:14:13 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 120.763828] nix-grpc-daemon[1382]: time=2026-09-23T05:14:13.216Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 120.765475] nix-grpc-daemon[1382]: time=2026-09-23T05:14:13.216Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)" lb # [ 120.682407] niks3-server[1111]: 2026/09/23 05:14:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign worker2 # [ 120.791100] nix-grpc-daemon[1382]: time=2026-09-23T05:14:13.244Z level=INFO msg="Uploading 1 narinfos" lb # [ 120.698469] niks3-server[1111]: 2026/09/23 05:14:13 INFO Signed narinfos id=41 count=1 lb # [ 120.705558] niks3-server[1111]: 2026/09/23 05:14:13 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls lb # [ 120.719433] niks3-server[1111]: 2026/09/23 05:14:13 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst lb # [ 120.726463] niks3-server[1111]: 2026/09/23 05:14:13 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete lb # [ 120.730747] niks3-server[1111]: 2026/09/23 05:14:13 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo worker2 # [ 120.837030] nix-grpc-daemon[1382]: time=2026-09-23T05:14:13.290Z level=INFO msg="Upload complete. (244ms)" lb # [ 120.738728] niks3-server[1111]: 2026/09/23 05:14:13 INFO Completed upload id=41 worker2 # [ 120.847449] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47682 duration_s=0 paths=1 nar_bytes_in=784 worker2 # [ 120.864121] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47682 worker2 # [ 120.870589] nix-daemon[855]: accepted connection from pid 1321, user nix-grpc-daemon (trusted) worker2 # [ 120.875872] nix-daemon[855]: reaped child process 1479, status = succeeded worker2 # [ 120.887481] nix-daemon[855]: accepted connection from pid 1321, user nix-grpc-daemon (trusted) client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051... client # farm-top> worker2: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-after-niks3-restart worker2 # [ 122.389281] nix-daemon[855]: reaped child process 1488, status = succeeded worker2 # [ 122.460621] nix-daemon[855]: accepted connection from pid 1504, user nix-grpc-daemon (trusted) worker2 # [ 122.482408] nix-daemon[855]: reaped child process 1507, status = succeeded lb # [ 122.379752] niks3-server[1111]: 2026/09/23 05:14:14 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 122.492371] nix-grpc-daemon[1382]: time=2026-09-23T05:14:14.944Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 122.493865] nix-grpc-daemon[1382]: time=2026-09-23T05:14:14.944Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)" lb # [ 122.423795] niks3-server[1111]: 2026/09/23 05:14:14 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst worker2 # [ 122.535228] nix-grpc-daemon[1382]: time=2026-09-23T05:14:14.988Z level=INFO msg="Uploading 1 narinfos" lb # [ 122.441380] niks3-server[1111]: 2026/09/23 05:14:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign lb # [ 122.451881] niks3-server[1111]: 2026/09/23 05:14:14 INFO Signed narinfos id=42 count=1 lb # [ 122.461732] niks3-server[1111]: 2026/09/23 05:14:14 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker2 # [ 122.572587] nix-grpc-daemon[1382]: time=2026-09-23T05:14:15.025Z level=INFO msg="Upload complete. (272ms)" lb # [ 122.471676] niks3-server[1111]: 2026/09/23 05:14:14 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls lb # [ 122.474931] niks3-server[1111]: 2026/09/23 05:14:14 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete lb # [ 122.481176] niks3-server[1111]: 2026/09/23 05:14:14 INFO Completed upload id=42 lb # [ 122.484354] niks3-server[1111]: 2026/09/23 05:14:14 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo worker2 # [ 122.583848] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47682 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=1 worker2 # [ 122.602595] nix-daemon[855]: reaped child process 1486, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2, in 6.86 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath, in 0.19 seconds) (finished: subtest: a niks3 restart does not cost the leader its role, in 7.62 seconds) subtest: scheduler down: the other node takes the lock and keeps it lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2 # [ 123.070391] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 123.112036] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:15Z level=info event=signal_exit lb # [ 123.251399] niks3-server[1111]: 2026/09/23 05:14:15 INFO lead: released remote=192.168.1.4:40674 worker1 # [ 124.281410] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:16Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 124.355503] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:16Z level=info event=scheduler_restarting worker2 # [ 124.360850] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:16Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 124.365525] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:16Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:56666 worker2 # [ 124.371579] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:16Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 124.380427] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:16Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:56666 worker2 # [ 124.388278] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:16Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 124.404371] nix-daemon[855]: reaped child process 1374, status = succeeded worker1 # [ 124.481245] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 124.487203] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 124.573497] nix-grpc-daemon[1321]: ts=2026-09-23T05:14:17Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker1 # [ 124.688338] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 124.694075] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 124.895134] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 124.900968] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 125.145152] nix-grpc-daemon[1321]: E0000 00:00:1790140457.597917 1367 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true worker2 # [ 125.152295] nix-grpc-daemon[1321]: E0000 00:00:1790140457.598248 1367 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker1 # [ 125.101975] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 125.107711] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 125.310821] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 125.315203] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 125.511880] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 125.517690] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 125.630012] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 125.636581] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 125.639235] systemd[1]: nix-grpc-daemon.service: Consumed 1.386s CPU time over 16.681s wall clock time, 39.3M memory peak, 45.9K incoming IP traffic, 78K outgoing IP traffic. worker2 # [ 125.659341] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over worker2 # [ 125.669746] systemd[1]: Closed Nix worker-protocol over gRPC. worker2 # [ 125.671360] systemd[1]: nix-grpc-daemon.socket: Consumed 894us CPU time over 17.776s wall clock time, 256K memory peak, 39.1K incoming IP traffic, 29.4K outgoing IP traffic. worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1 # [ 125.719814] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 125.722515] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds) worker1 # [ 125.921325] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 125.926882] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 126.127712] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 126.133243] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 126.266429] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:18Z level=info event=scheduler_take_over lb # [ 126.235042] niks3-server[1111]: 2026/09/23 05:14:18 INFO lead: acquired remote=192.168.1.3:60566 worker1 # [ 126.335879] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 126.342123] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 126.536941] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 126.546323] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:19Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:49278 worker1 # [ 126.552305] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:19Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2 worker1 # [ 127.015005] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:45390 client # don't know how to build these paths: client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker1 # [ 127.025860] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 127.054400] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:19Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=0 worker1 # [ 127.062202] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:19Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 127.100875] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 127.117316] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:45390 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'... worker1 # [ 127.134298] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 127.150013] nix-daemon[832]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) worker1 # [ 127.177966] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:19Z level=info event=niks3_push_started pid=1232 worker1 # [ 127.330459] nix-grpc-daemon[1232]: time=2026-09-23T05:14:19.836Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 127.446091] nix-daemon[832]: accepted connection from pid 1245, user nix-grpc-daemon (trusted) worker1 # [ 127.467549] nix-daemon[832]: reaped child process 1248, status = succeeded lb # [ 127.441640] niks3-server[1111]: 2026/09/23 05:14:19 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 127.484078] nix-grpc-daemon[1232]: time=2026-09-23T05:14:19.989Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 127.488616] nix-grpc-daemon[1232]: time=2026-09-23T05:14:19.989Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)" lb # [ 127.493106] niks3-server[1111]: 2026/09/23 05:14:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign worker1 # [ 127.532065] nix-grpc-daemon[1232]: time=2026-09-23T05:14:20.037Z level=INFO msg="Uploading 1 narinfos" lb # [ 127.505049] niks3-server[1111]: 2026/09/23 05:14:20 INFO Signed narinfos id=43 count=1 lb # [ 127.510705] niks3-server[1111]: 2026/09/23 05:14:20 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst lb # [ 127.522039] niks3-server[1111]: 2026/09/23 05:14:20 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls lb # [ 127.533819] niks3-server[1111]: 2026/09/23 05:14:20 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete worker1 # [ 127.570612] nix-grpc-daemon[1232]: time=2026-09-23T05:14:20.077Z level=INFO msg="Upload complete. (236ms)" lb # [ 127.541384] niks3-server[1111]: 2026/09/23 05:14:20 INFO Completed upload id=43 lb # [ 127.543325] niks3-server[1111]: 2026/09/23 05:14:20 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo worker1 # [ 127.588224] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45390 duration_s=0 paths=1 nar_bytes_in=520 worker1 # [ 127.599537] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 127.606097] nix-daemon[832]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) worker1 # [ 127.611045] nix-daemon[832]: reaped child process 1230, status = succeeded client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051... worker1 # [ 127.623408] nix-daemon[832]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) client # farm-a> node-a: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-on-standby worker1 # [ 128.958359] nix-daemon[832]: reaped child process 1252, status = succeeded worker1 # [ 129.050203] nix-daemon[832]: accepted connection from pid 1268, user nix-grpc-daemon (trusted) worker1 # [ 129.077665] nix-daemon[832]: reaped child process 1271, status = succeeded lb # [ 129.045188] niks3-server[1111]: 2026/09/23 05:14:21 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 129.092563] nix-grpc-daemon[1232]: time=2026-09-23T05:14:21.596Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 129.094294] nix-grpc-daemon[1232]: time=2026-09-23T05:14:21.596Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)" lb # [ 129.103350] niks3-server[1111]: 2026/09/23 05:14:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign lb # [ 129.116686] niks3-server[1111]: 2026/09/23 05:14:21 INFO Signed narinfos id=44 count=1 worker1 # [ 129.150140] nix-grpc-daemon[1232]: time=2026-09-23T05:14:21.655Z level=INFO msg="Uploading 1 narinfos" lb # [ 129.131414] niks3-server[1111]: 2026/09/23 05:14:21 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv lb # [ 129.143046] niks3-server[1111]: 2026/09/23 05:14:21 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst lb # [ 129.153731] niks3-server[1111]: 2026/09/23 05:14:21 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls lb # [ 129.167153] niks3-server[1111]: 2026/09/23 05:14:21 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete worker1 # [ 129.204436] nix-grpc-daemon[1232]: time=2026-09-23T05:14:21.710Z level=INFO msg="Upload complete. (364ms)" lb # [ 129.172807] niks3-server[1111]: 2026/09/23 05:14:21 INFO Completed upload id=44 lb # [ 129.183606] niks3-server[1111]: 2026/09/23 05:14:21 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo lb # [ 129.189869] envoy[895]: conn peer=[2001:db8:1::1]:60912 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9792 tx=4446 ms=6597 lb # [ 129.194451] envoy[895]: conn peer=[2001:db8:1::4]:35706 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=1708 tx=442 ms=15392 worker1 # [ 129.234517] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45390 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=1 worker1 # [ 129.248660] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:21Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=0 worker1 # [ 129.257504] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:21Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 129.261871] nix-daemon[832]: reaped child process 1250, status = succeeded worker1 # [ 129.286386] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 129.307945] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 129.325785] nix-grpc-daemon[1232]: time=2026-09-23T05:14:21.832Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker1 # [ 129.340195] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45390 client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'... worker1 # [ 129.362801] nix-daemon[832]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) worker1 # [ 129.558370] nix-daemon[832]: accepted connection from pid 1277, user nix-grpc-daemon (trusted) worker1 # [ 129.583840] nix-daemon[832]: reaped child process 1280, status = succeeded lb # [ 129.550105] niks3-server[1111]: 2026/09/23 05:14:22 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 129.592920] nix-grpc-daemon[1232]: time=2026-09-23T05:14:22.099Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 129.594472] nix-grpc-daemon[1232]: time=2026-09-23T05:14:22.099Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)" lb # [ 129.585133] niks3-server[1111]: 2026/09/23 05:14:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign worker1 # [ 129.629381] nix-grpc-daemon[1232]: time=2026-09-23T05:14:22.135Z level=INFO msg="Uploading 1 narinfos" lb # [ 129.603210] niks3-server[1111]: 2026/09/23 05:14:22 INFO Signed narinfos id=45 count=1 lb # [ 129.616278] niks3-server[1111]: 2026/09/23 05:14:22 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls lb # [ 129.627464] niks3-server[1111]: 2026/09/23 05:14:22 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst lb # [ 129.643577] niks3-server[1111]: 2026/09/23 05:14:22 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete worker1 # [ 129.681409] nix-grpc-daemon[1232]: time=2026-09-23T05:14:22.187Z level=INFO msg="Upload complete. (298ms)" lb # [ 129.653086] niks3-server[1111]: 2026/09/23 05:14:22 INFO Completed upload id=45 lb # [ 129.655707] niks3-server[1111]: 2026/09/23 05:14:22 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo worker1 # [ 129.692446] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45390 duration_s=0 paths=1 nar_bytes_in=640 worker1 # [ 129.708928] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 129.727462] nix-daemon[832]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) worker1 # [ 129.734160] nix-daemon[832]: reaped child process 1275, status = succeeded client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051... worker1 # [ 129.749829] nix-daemon[832]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) client # farm-b> node-a: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-on-standby worker1 # [ 131.086080] nix-daemon[832]: reaped child process 1284, status = succeeded worker1 # [ 131.173287] nix-daemon[832]: accepted connection from pid 1300, user nix-grpc-daemon (trusted) worker1 # [ 131.198167] nix-daemon[832]: reaped child process 1303, status = succeeded lb # [ 131.166523] niks3-server[1111]: 2026/09/23 05:14:23 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 131.216313] nix-grpc-daemon[1232]: time=2026-09-23T05:14:23.721Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 131.217971] nix-grpc-daemon[1232]: time=2026-09-23T05:14:23.721Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)" lb # [ 131.230073] niks3-server[1111]: 2026/09/23 05:14:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign worker1 # [ 131.271319] nix-grpc-daemon[1232]: time=2026-09-23T05:14:23.777Z level=INFO msg="Uploading 1 narinfos" lb # [ 131.247048] niks3-server[1111]: 2026/09/23 05:14:23 INFO Signed narinfos id=46 count=1 lb # [ 131.259055] niks3-server[1111]: 2026/09/23 05:14:23 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst lb # [ 131.269743] niks3-server[1111]: 2026/09/23 05:14:23 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv lb # [ 131.275398] niks3-server[1111]: 2026/09/23 05:14:23 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls lb # [ 131.283960] niks3-server[1111]: 2026/09/23 05:14:23 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete lb # [ 131.291725] niks3-server[1111]: 2026/09/23 05:14:23 INFO Completed upload id=46 worker1 # [ 131.324319] nix-grpc-daemon[1232]: time=2026-09-23T05:14:23.830Z level=INFO msg="Upload complete. (354ms)" lb # [ 131.294777] niks3-server[1111]: 2026/09/23 05:14:23 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo worker1 # [ 131.348164] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45390 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=1 worker1 # [ 131.372555] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:23Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=0 worker1 # [ 131.377697] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:23Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 131.385027] nix-daemon[832]: reaped child process 1282, status = succeeded worker1 # [ 131.398623] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 131.419607] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 131.439025] nix-grpc-daemon[1232]: time=2026-09-23T05:14:23.945Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'... worker1 # [ 131.455091] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 131.474806] nix-daemon[832]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) worker1 # [ 131.669966] nix-daemon[832]: accepted connection from pid 1309, user nix-grpc-daemon (trusted) worker1 # [ 131.694541] nix-daemon[832]: reaped child process 1312, status = succeeded lb # [ 131.661131] niks3-server[1111]: 2026/09/23 05:14:24 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 131.705690] nix-grpc-daemon[1232]: time=2026-09-23T05:14:24.211Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 131.707206] nix-grpc-daemon[1232]: time=2026-09-23T05:14:24.211Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)" lb # [ 131.709774] niks3-server[1111]: 2026/09/23 05:14:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign worker1 # [ 131.749058] nix-grpc-daemon[1232]: time=2026-09-23T05:14:24.255Z level=INFO msg="Uploading 1 narinfos" lb # [ 131.724576] niks3-server[1111]: 2026/09/23 05:14:24 INFO Signed narinfos id=47 count=1 lb # [ 131.730333] niks3-server[1111]: 2026/09/23 05:14:24 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst lb # [ 131.743641] niks3-server[1111]: 2026/09/23 05:14:24 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls lb # [ 131.758325] niks3-server[1111]: 2026/09/23 05:14:24 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete lb # [ 131.767995] niks3-server[1111]: 2026/09/23 05:14:24 INFO Completed upload id=47 worker1 # [ 131.799493] nix-grpc-daemon[1232]: time=2026-09-23T05:14:24.305Z level=INFO msg="Upload complete. (301ms)" lb # [ 131.772452] niks3-server[1111]: 2026/09/23 05:14:24 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo worker1 # [ 131.810754] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45390 duration_s=0 paths=1 nar_bytes_in=768 worker1 # [ 131.833933] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 131.840467] nix-daemon[832]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) worker1 # [ 131.846125] nix-daemon[832]: reaped child process 1307, status = succeeded client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051... client # farm-top> node-a: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker1 # [ 131.858526] nix-daemon[832]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-on-standby worker1 # [ 132.533599] nix-daemon[832]: reaped child process 1316, status = succeeded worker1 # [ 132.600100] nix-daemon[832]: accepted connection from pid 1332, user nix-grpc-daemon (trusted) worker1 # [ 132.618633] nix-daemon[832]: reaped child process 1335, status = succeeded lb # [ 132.593983] niks3-server[1111]: 2026/09/23 05:14:25 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 132.636405] nix-grpc-daemon[1232]: time=2026-09-23T05:14:25.141Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 132.637899] nix-grpc-daemon[1232]: time=2026-09-23T05:14:25.141Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)" lb # [ 132.638519] niks3-server[1111]: 2026/09/23 05:14:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign worker1 # [ 132.678118] nix-grpc-daemon[1232]: time=2026-09-23T05:14:25.184Z level=INFO msg="Uploading 1 narinfos" lb # [ 132.653456] niks3-server[1111]: 2026/09/23 05:14:25 INFO Signed narinfos id=48 count=1 lb # [ 132.659857] niks3-server[1111]: 2026/09/23 05:14:25 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv lb # [ 132.667746] niks3-server[1111]: 2026/09/23 05:14:25 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst lb # [ 132.676185] niks3-server[1111]: 2026/09/23 05:14:25 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls lb # [ 132.679964] niks3-server[1111]: 2026/09/23 05:14:25 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete worker1 # [ 132.711743] nix-grpc-daemon[1232]: time=2026-09-23T05:14:25.218Z level=INFO msg="Upload complete. (241ms)" lb # [ 132.683950] niks3-server[1111]: 2026/09/23 05:14:25 INFO Completed upload id=48 lb # [ 132.687351] niks3-server[1111]: 2026/09/23 05:14:25 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo worker1 # [ 132.728496] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45390 duration_s=0 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=1 worker1 # [ 132.740413] nix-daemon[832]: reaped child process 1314, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 5.98 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.18 seconds) worker2 # [ 133.117065] systemd[1]: Listening on Nix worker-protocol over gRPC. worker2 # [ 133.123948] systemd[1]: Starting scheduler-token.service... worker2 # [ 133.199022] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 133.203428] systemd[1]: Finished scheduler-token.service. worker2 # [ 133.208615] systemd[1]: scheduler-token.service: Consumed 43ms CPU time over 78ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 133.220818] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 133.336197] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:25Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 133.338462] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:25Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 133.342526] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:25Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 133.348052] nix-grpc-daemon[1528]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 133.349483] nix-grpc-daemon[1528]: E0000 00:00:1790140465.800295 1528 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 133.392250] nix-grpc-daemon[1528]: E0000 00:00:1790140465.806621 1528 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 133.394521] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:25Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 133.398988] nix-grpc-daemon[1528]: E0000 00:00:1790140465.814760 1568 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 133.401935] nix-grpc-daemon[1528]: E0000 00:00:1790140465.829933 1568 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 133.404424] nix-grpc-daemon[1528]: E0000 00:00:1790140465.830124 1568 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 133.406796] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 133.407604] nix-grpc-daemon[1528]: E0000 00:00:1790140465.851027 1568 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 133.417253] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 133.358288] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:25Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 133.362950] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:25Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:49278 worker1 # [ 133.368472] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:25Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: scheduler down: the other node takes the lock and keeps it, in 10.89 seconds) subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs client # [ 133.736803] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs. client # Running as unit: rs.service; invocation ID: ae2c02fa55c9441a8a1a5b9ecf05ec58 client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.09 seconds) worker2 # [ 133.991328] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:56074 worker2 # [ 134.006375] nix-daemon[855]: accepted connection from pid 1528, user nix-grpc-daemon (trusted) client # [ 133.976137] nix[1418]: don't know how to build these paths: client # [ 133.977752] nix[1418]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker1 # [ 133.971886] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:26Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=0 worker1 # [ 133.978919] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:26Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 134.017262] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 134.033132] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:45390 client # [ 134.068564] nix[1418]: copying 1 paths to 'lb:50051'... worker1 # [ 134.048124] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45390 client # [ 134.070967] nix[1418]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'... worker1 # [ 134.066364] nix-daemon[832]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) worker1 # [ 134.225758] nix-daemon[832]: accepted connection from pid 1348, user nix-grpc-daemon (trusted) worker1 # [ 134.248559] nix-daemon[832]: reaped child process 1351, status = succeeded lb # [ 134.218848] niks3-server[1111]: 2026/09/23 05:14:26 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 134.264880] nix-grpc-daemon[1232]: time=2026-09-23T05:14:26.770Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 134.266532] nix-grpc-daemon[1232]: time=2026-09-23T05:14:26.770Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)" lb # [ 134.252127] niks3-server[1111]: 2026/09/23 05:14:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign worker1 # [ 134.290325] nix-grpc-daemon[1232]: time=2026-09-23T05:14:26.796Z level=INFO msg="Uploading 1 narinfos" lb # [ 134.260439] niks3-server[1111]: 2026/09/23 05:14:26 INFO Signed narinfos id=49 count=1 lb # [ 134.268753] niks3-server[1111]: 2026/09/23 05:14:26 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls lb # [ 134.275533] niks3-server[1111]: 2026/09/23 05:14:26 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst lb # [ 134.285361] niks3-server[1111]: 2026/09/23 05:14:26 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete lb # [ 134.292749] niks3-server[1111]: 2026/09/23 05:14:26 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo worker1 # [ 134.324504] nix-grpc-daemon[1232]: time=2026-09-23T05:14:26.831Z level=INFO msg="Upload complete. (241ms)" lb # [ 134.295527] niks3-server[1111]: 2026/09/23 05:14:26 INFO Completed upload id=49 worker1 # [ 134.332379] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45390 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 134.351185] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45390 worker1 # [ 134.358930] nix-daemon[832]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) worker1 # [ 134.364867] nix-daemon[832]: reaped child process 1346, status = succeeded worker1 # [ 134.378109] nix-daemon[832]: accepted connection from pid 1155, user nix-grpc-daemon (trusted) client # [ 134.397090] nix[1418]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051... lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true worker2 # [ 136.598449] systemd-journald[269]: Received client request to rotate journal, rotating. worker2 # [ 136.621992] systemd-journald[269]: Vacuuming done, freed 0B of archived journals from /var/log/journal/d2d4cc527b124966b5a15337f83eea32. worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.09 seconds) worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service worker1 # [ 136.592882] systemd[1]: Stopping Nix worker-protocol over gRPC... worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds) client: must succeed: sleep 5.0 worker1 # [ 136.966808] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:29Z level=info event=draining lb # [ 139.188737] envoy[895]: conn peer=[2001:db8:1::1]:54444 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9580 tx=4368 ms=5723 client: (finished: must succeed: sleep 5.0, in 5.05 seconds) worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=' worker1 # [ 142.088493] nix-daemon[832]: reaped child process 1355, status = succeeded worker1 # [ 142.163517] nix-daemon[832]: accepted connection from pid 1396, user nix-grpc-daemon (trusted) worker1 # [ 142.183954] nix-daemon[832]: reaped child process 1399, status = succeeded lb # [ 142.158508] niks3-server[1111]: 2026/09/23 05:14:34 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 142.202949] nix-grpc-daemon[1232]: time=2026-09-23T05:14:34.706Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 142.204967] nix-grpc-daemon[1232]: time=2026-09-23T05:14:34.706Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)" lb # [ 142.200743] niks3-server[1111]: 2026/09/23 05:14:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign worker1 # [ 142.242978] nix-grpc-daemon[1232]: time=2026-09-23T05:14:34.749Z level=INFO msg="Uploading 1 narinfos" lb # [ 142.215494] niks3-server[1111]: 2026/09/23 05:14:34 INFO Signed narinfos id=50 count=1 lb # [ 142.222856] niks3-server[1111]: 2026/09/23 05:14:34 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls lb # [ 142.230466] niks3-server[1111]: 2026/09/23 05:14:34 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst lb # [ 142.238756] niks3-server[1111]: 2026/09/23 05:14:34 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker1 # [ 142.275831] nix-grpc-daemon[1232]: time=2026-09-23T05:14:34.782Z level=INFO msg="Upload complete. (279ms)" lb # [ 142.245112] niks3-server[1111]: 2026/09/23 05:14:34 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete lb # [ 142.249506] niks3-server[1111]: 2026/09/23 05:14:34 INFO Completed upload id=50 lb # [ 142.253276] niks3-server[1111]: 2026/09/23 05:14:34 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo worker1 # [ 142.292351] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45390 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=1 worker1 # [ 142.309836] nix-daemon[832]: reaped child process 1353, status = succeeded client # [ 142.345970] systemd[1]: rs.service: Deactivated successfully. client # [ 142.351275] systemd[1]: rs.service: Consumed 196ms CPU time over 8.610s wall clock time, 41.4M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic. worker1 # [ 142.967871] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:35Z level=info event=signal_exit lb # [ 143.245012] niks3-server[1111]: 2026/09/23 05:14:35 INFO lead: released remote=192.168.1.3:60566 lb # [ 143.258676] niks3-server[1111]: 2026/09/23 05:14:35 INFO lead: acquired remote=192.168.1.4:48734 worker2 # [ 143.365984] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:35Z level=info event=scheduler_take_over worker2 # [ 144.340271] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:36Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 144.276896] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:36Z level=info event=scheduler_restarting worker1 # [ 144.280823] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:36Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 144.285391] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:36Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:49278 worker1 # [ 144.290306] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:36Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 144.294710] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:36Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:49278 worker1 # [ 144.300192] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:36Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 144.323964] nix-daemon[832]: reaped child process 1208, status = succeeded worker1 # [ 144.388206] nix-grpc-daemon[1155]: E0000 00:00:1790140476.894377 1201 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true worker1 # [ 144.394830] nix-grpc-daemon[1155]: E0000 00:00:1790140476.894723 1201 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true worker2 # [ 144.543034] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 144.548377] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 144.493657] nix-grpc-daemon[1155]: ts=2026-09-23T05:14:36Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 144.749622] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 144.767279] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:37Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 144.771789] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:37Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:47566 worker2 # [ 144.777243] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:37Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.24 seconds) client: waiting for success: ! systemctl is-active rs client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds) client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; } client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds) client: must fail: journalctl -u rs | grep -q 'reconnecting' client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker1 # [ 145.522119] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 145.526921] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 145.531491] systemd[1]: nix-grpc-daemon.service: Consumed 1.786s CPU time over 31.748s wall clock time, 38.8M memory peak, 47.9K incoming IP traffic, 96K outgoing IP traffic. worker1 # [ 145.552338] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 145.698041] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:38Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 145.700409] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:38Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 145.704458] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:38Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 145.711222] nix-grpc-daemon[1406]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 145.728169] nix-grpc-daemon[1406]: E0000 00:00:1790140478.215560 1406 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 145.730576] nix-grpc-daemon[1406]: E0000 00:00:1790140478.217706 1406 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 145.754998] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:38Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 145.757568] nix-grpc-daemon[1406]: E0000 00:00:1790140478.225844 1446 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 145.761098] nix-grpc-daemon[1406]: E0000 00:00:1790140478.246180 1446 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 145.766500] nix-grpc-daemon[1406]: E0000 00:00:1790140478.246337 1446 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 145.771035] nix-grpc-daemon[1406]: E0000 00:00:1790140478.260762 1446 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 145.773718] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 145.774563] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 145.842663] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:38Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:47566 worker2 # [ 145.848477] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:38Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.91 seconds) subtest: scheduler restart mid-build: build finishes, no double build client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid client # [ 146.644817] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid. client # Running as unit: mid.service; invocation ID: d68625aa1e1b4250a53f1101f4f0e1ba client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.09 seconds) worker2 # [ 146.897051] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:39Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:56074 client # [ 146.855205] nix[1454]: don't know how to build these paths: client # [ 146.856342] nix[1454]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker2 # [ 146.917525] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:39Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker2 # [ 146.924631] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:39Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 146.961787] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56074 worker2 # [ 146.978130] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 20 ms (attempt 1/5) worker2 # [ 147.007328] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 95 ms (attempt 2/5) worker2 # [ 147.104108] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 310 ms (attempt 3/5) worker2 # [ 147.416095] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 644 ms (attempt 4/5) worker2 # [ 148.064784] nix-grpc-daemon[1528]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 148.068647] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 148.082668] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56074 client # [ 148.053566] nix[1454]: copying 1 paths to 'lb:50051'... worker2 # [ 148.103685] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56074 client # [ 148.056696] nix[1454]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'... worker2 # [ 148.122123] nix-daemon[855]: accepted connection from pid 1528, user nix-grpc-daemon (trusted) worker2 # [ 148.148104] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:40Z level=info event=niks3_push_started pid=1661 worker2 # [ 148.298881] nix-grpc-daemon[1661]: time=2026-09-23T05:14:40.751Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 148.404456] nix-daemon[855]: accepted connection from pid 1673, user nix-grpc-daemon (trusted) worker2 # [ 148.422836] nix-daemon[855]: reaped child process 1676, status = succeeded lb # [ 148.326880] niks3-server[1111]: 2026/09/23 05:14:40 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 148.438945] nix-grpc-daemon[1661]: time=2026-09-23T05:14:40.891Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 148.442839] nix-grpc-daemon[1661]: time=2026-09-23T05:14:40.891Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)" lb # [ 148.374195] niks3-server[1111]: 2026/09/23 05:14:40 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls worker2 # [ 148.485690] nix-grpc-daemon[1661]: time=2026-09-23T05:14:40.938Z level=INFO msg="Uploading 1 narinfos" lb # [ 148.385837] niks3-server[1111]: 2026/09/23 05:14:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign lb # [ 148.395241] niks3-server[1111]: 2026/09/23 05:14:40 INFO Signed narinfos id=51 count=1 lb # [ 148.400988] niks3-server[1111]: 2026/09/23 05:14:40 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst lb # [ 148.412731] niks3-server[1111]: 2026/09/23 05:14:40 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete worker2 # [ 148.520676] nix-grpc-daemon[1661]: time=2026-09-23T05:14:40.973Z level=INFO msg="Upload complete. (218ms)" lb # [ 148.421436] niks3-server[1111]: 2026/09/23 05:14:40 INFO Completed upload id=51 worker2 # [ 148.528137] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56074 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 148.540594] nix-grpc-daemon[1528]: ts=2026-09-23T05:14:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56074 worker2 # [ 148.551520] nix-daemon[855]: accepted connection from pid 1528, user nix-grpc-daemon (trusted) worker2 # [ 148.556897] nix-daemon[855]: reaped child process 1659, status = succeeded worker2 # [ 148.568151] nix-daemon[855]: accepted connection from pid 1528, user nix-grpc-daemon (trusted) client # [ 148.516548] nix[1454]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... lb # [ 149.190048] envoy[895]: conn peer=[2001:db8:1::1]:52254 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2966 tx=1517 ms=8392 lb # [ 149.195284] envoy[895]: conn peer=[2001:db8:1::3]:56932 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2852 tx=977 ms=30475 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service worker2 # [ 150.572439] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1528 (nix-grpc-daemon) on client request. worker2 # [ 150.574049] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request. lb # [ 150.491675] niks3-server[1111]: 2026/09/23 05:14:42 INFO lead: released remote=192.168.1.4:48734 worker2 # [ 150.597148] systemd[1]: Starting scheduler-token.service... client # [ 150.544926] nix[1454]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again worker1 # [ 150.524732] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:43Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination" client # [ 150.551282] nix[1454]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting worker2 # [ 150.608452] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL worker2 # [ 150.609943] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'. worker2 # [ 150.611285] systemd[1]: nix-grpc-daemon.service: Consumed 496ms CPU time over 17.391s wall clock time, 34.7M memory peak, 15.3K incoming IP traffic, 25.6K outgoing IP traffic. worker2 # [ 150.655478] nix-daemon[855]: reaped child process 1678, status = succeeded worker2 # [ 150.657596] nix-daemon[855]: reaped child process 1588, status = succeeded worker2 # [ 150.664500] nix-daemon[1680]: unexpected Nix daemon error: error: interrupted by the user worker2 # [ 150.676600] nix-daemon[855]: reaped child process 1680, status = failed with exit code 1 worker2 # [ 150.693441] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 150.694679] systemd[1]: Finished scheduler-token.service. worker2 # [ 150.695797] systemd[1]: scheduler-token.service: Consumed 24ms CPU time over 93ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 150.699729] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1. worker2 # [ 150.704260] systemd[1]: Starting scheduler-token.service... worker2 # [ 150.745292] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 150.748139] systemd[1]: Finished scheduler-token.service. worker2 # [ 150.748961] systemd[1]: scheduler-token.service: Consumed 22ms CPU time over 47ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 150.753294] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 150.689538] niks3-server[1111]: 2026/09/23 05:14:43 INFO lead: acquired remote=192.168.1.3:40566 worker1 # [ 150.724728] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 150.730096] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:43Z level=info event=scheduler_take_over worker2 # [ 150.853751] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:43Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 150.855983] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:43Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 150.859803] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:43Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 150.865275] nix-grpc-daemon[1723]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 150.866716] nix-grpc-daemon[1723]: E0000 00:00:1790140483.317737 1723 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.36 seconds) client: must succeed: sleep 5.0 worker1 # [ 150.820566] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:43Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker2 # [ 150.904153] nix-grpc-daemon[1723]: E0000 00:00:1790140483.323613 1723 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 150.906389] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:43Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 150.907977] nix-grpc-daemon[1723]: E0000 00:00:1790140483.338806 1765 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 150.926371] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 150.927642] nix-grpc-daemon[1723]: E0000 00:00:1790140483.364352 1765 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 150.930247] nix-grpc-daemon[1723]: E0000 00:00:1790140483.364484 1765 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 150.935167] nix-grpc-daemon[1723]: E0000 00:00:1790140483.378939 1765 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 150.939848] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 150.941262] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 151.021487] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:43Z level=info event=scheduler_connected addr=lb:50051 client # [ 151.047903] nix[1454]: scheduler at lb:50051: no healthy upstream, reconnecting worker1 # [ 151.026826] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 151.141392] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 151.146266] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 151.227674] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 151.232991] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 151.347274] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 151.352421] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 151.433712] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 151.438863] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 151.553317] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 151.558332] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 151.641370] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 151.645549] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 151.759315] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 151.764448] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 151.841879] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 151.847087] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 151.967249] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 151.971408] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" client # [ 152.051994] nix[1454]: scheduler at lb:50051: no healthy upstream, reconnecting worker1 # [ 152.047878] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 152.053176] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 152.167923] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 152.173009] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 152.253743] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 152.258852] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 152.374023] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 152.379087] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 152.459728] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 152.465058] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 152.579873] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 152.585044] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 152.666995] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 152.686527] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:45Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:38376 worker1 # [ 152.692827] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:45Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker2 # [ 152.785849] nix-grpc-daemon[1723]: ts=2026-09-23T05:14:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 152.725787] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:45Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 152.730250] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:45Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:38376 worker1 # [ 152.735792] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:45Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 154.044515] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:46Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker1 # [ 154.049911] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:46Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 154.101948] nix-grpc-daemon[1406]: ts=2026-09-23T05:14:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38380 worker1 # [ 154.118522] nix-daemon[832]: accepted connection from pid 1406, user nix-grpc-daemon (trusted) worker1 # [ 154.141702] nix-grpc-daemon[1406]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 18 ms (attempt 1/5) worker1 # [ 154.166830] nix-grpc-daemon[1406]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 40 ms (attempt 2/5) worker1 # [ 154.208575] nix-grpc-daemon[1406]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 372 ms (attempt 3/5) worker1 # [ 154.582583] nix-grpc-daemon[1406]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 185 ms (attempt 4/5) worker1 # [ 154.770644] nix-grpc-daemon[1406]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 154.776434] nix-grpc-daemon[1406]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 154.793530] nix-grpc-daemon[1406]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' from 'http://lb:5751'... worker1 # [ 154.821302] nix-daemon[832]: accepted connection from pid 1406, user nix-grpc-daemon (trusted) client # [ 154.859599] nix[1454]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... worker1 # [ 154.843292] nix-daemon[832]: accepted connection from pid 1406, user nix-grpc-daemon (trusted) client: (finished: must succeed: sleep 5.0, in 5.05 seconds) client: waiting for success: ! systemctl is-active mid lb # [ 159.190362] envoy[895]: conn peer=[2001:db8:1::4]:47436 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=2696 tx=393 ms=17189 worker1 # [ 246.342152] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:18Z level=info event=niks3_push_started pid=1515 worker1 # [ 246.449189] nix-daemon[832]: reaped child process 1490, status = succeeded worker1 # [ 246.522976] nix-grpc-daemon[1515]: time=2026-09-23T05:16:19.029Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 246.635918] nix-daemon[832]: accepted connection from pid 1527, user nix-grpc-daemon (trusted) worker1 # [ 246.653066] nix-daemon[832]: reaped child process 1530, status = succeeded lb # [ 246.629887] niks3-server[1111]: 2026/09/23 05:16:19 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 246.681645] nix-grpc-daemon[1515]: time=2026-09-23T05:16:19.186Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 246.683133] nix-grpc-daemon[1515]: time=2026-09-23T05:16:19.186Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)" lb # [ 246.707273] niks3-server[1111]: 2026/09/23 05:16:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign worker1 # [ 246.748140] nix-grpc-daemon[1515]: time=2026-09-23T05:16:19.253Z level=INFO msg="Uploading 1 narinfos" lb # [ 246.720194] niks3-server[1111]: 2026/09/23 05:16:19 INFO Signed narinfos id=52 count=1 lb # [ 246.731946] niks3-server[1111]: 2026/09/23 05:16:19 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv lb # [ 246.739771] niks3-server[1111]: 2026/09/23 05:16:19 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst lb # [ 246.745967] niks3-server[1111]: 2026/09/23 05:16:19 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls lb # [ 246.753825] niks3-server[1111]: 2026/09/23 05:16:19 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete lb # [ 246.759481] niks3-server[1111]: 2026/09/23 05:16:19 INFO Completed upload id=52 worker1 # [ 246.791333] nix-grpc-daemon[1515]: time=2026-09-23T05:16:19.297Z level=INFO msg="Upload complete. (264ms)" lb # [ 246.761854] niks3-server[1111]: 2026/09/23 05:16:19 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo worker1 # [ 246.808416] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38380 duration_s=92 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=1 worker1 # [ 246.829274] nix-daemon[832]: reaped child process 1488, status = succeeded client # [ 246.874874] systemd[1]: mid.service: Deactivated successfully. client # [ 246.880801] systemd[1]: mid.service: Consumed 285ms CPU time over 1min 40.235s wall clock time, 41.2M memory peak, 4K written to disk, 10.8K incoming IP traffic, 12.2K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active mid, in 91.46 seconds) client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; } client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.05 seconds) (finished: subtest: scheduler restart mid-build: build finishes, no double build, in 101.13 seconds) subtest: interrupting the client stops the build on the worker client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr client # [ 247.744973] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr. client # Running as unit: intr.service; invocation ID: bfe621e9eafc4e99afa39235b41c06e1 client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.05 seconds) client # [ 247.932755] nix[1915]: don't know how to build these paths: client # [ 247.933753] nix[1915]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker1 # [ 247.912370] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:20Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:38380 worker1 # [ 247.927388] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:20Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=0 worker1 # [ 247.933489] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:20Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 248.061797] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 248.081104] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 248.111572] nix-grpc-daemon[1723]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 67 ms (attempt 1/5) worker2 # [ 248.186564] nix-grpc-daemon[1723]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 16 ms (attempt 2/5) worker2 # [ 248.211578] nix-grpc-daemon[1723]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 333 ms (attempt 3/5) worker2 # [ 248.603801] nix-grpc-daemon[1723]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 561 ms (attempt 4/5) worker2 # [ 249.169706] nix-grpc-daemon[1723]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 249.176879] nix-grpc-daemon[1723]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 249.195774] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47332 client # [ 249.162245] nix[1915]: copying 1 paths to 'lb:50051'... client # [ 249.166961] nix[1915]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'... worker2 # [ 249.217234] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 249.234230] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 249.260822] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:21Z level=info event=niks3_push_started pid=1805 lb # [ 249.198078] envoy[895]: conn peer=[2001:db8:1::1]:45116 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4480 tx=2190 ms=99994 worker2 # [ 249.434091] nix-grpc-daemon[1805]: time=2026-09-23T05:16:21.884Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 249.539391] nix-daemon[855]: accepted connection from pid 1818, user nix-grpc-daemon (trusted) worker2 # [ 249.557375] nix-daemon[855]: reaped child process 1821, status = succeeded lb # [ 249.462465] niks3-server[1111]: 2026/09/23 05:16:21 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 249.576067] nix-grpc-daemon[1805]: time=2026-09-23T05:16:22.028Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 249.577542] nix-grpc-daemon[1805]: time=2026-09-23T05:16:22.028Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)" lb # [ 249.504409] niks3-server[1111]: 2026/09/23 05:16:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign worker2 # [ 249.622365] nix-grpc-daemon[1805]: time=2026-09-23T05:16:22.074Z level=INFO msg="Uploading 1 narinfos" lb # [ 249.520383] niks3-server[1111]: 2026/09/23 05:16:22 INFO Signed narinfos id=53 count=1 lb # [ 249.529936] niks3-server[1111]: 2026/09/23 05:16:22 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls lb # [ 249.548781] niks3-server[1111]: 2026/09/23 05:16:22 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete worker2 # [ 249.663706] nix-grpc-daemon[1805]: time=2026-09-23T05:16:22.116Z level=INFO msg="Upload complete. (230ms)" lb # [ 249.562608] niks3-server[1111]: 2026/09/23 05:16:22 INFO Completed upload id=53 lb # [ 249.570838] niks3-server[1111]: 2026/09/23 05:16:22 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo worker2 # [ 249.673499] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=0 paths=1 nar_bytes_in=424 lb # [ 249.578467] niks3-server[1111]: 2026/09/23 05:16:22 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst worker2 # [ 249.689723] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 249.696265] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 249.701397] nix-daemon[855]: reaped child process 1803, status = succeeded worker2 # [ 249.713499] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) client # [ 249.662024] nix[1915]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051... client: must succeed: systemctl kill -s INT intr client # [ 251.405432] systemd[1]: intr.service: Sent signal SIGINT to main process 1915 (nix) on client request. worker2 # [ 251.465512] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:23Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv client # [ 251.416741] nix[1915]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client # [ 251.428374] nix[1915]: error: interrupted by the user client: (finished: must succeed: systemctl kill -s INT intr, in 0.08 seconds) client # [ 251.450840] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE client # [ 251.455334] systemd[1]: intr.service: Failed with result 'exit-code'. client # [ 251.457361] systemd[1]: intr.service: Consumed 182ms CPU time over 3.708s wall clock time, 41.4M memory peak, 6.4K incoming IP traffic, 8.3K outgoing IP traffic. worker2 # [ 251.540599] nix-daemon[1825]: unexpected Nix daemon error: error: interrupted by the user worker2 # [ 251.550127] nix-daemon[855]: reaped child process 1823, status = succeeded worker2 # [ 251.551479] nix-daemon[855]: reaped child process 1825, status = failed with exit code 1 (finished: subtest: interrupting the client stops the build on the worker, in 3.82 seconds) subtest: a deploy mid-build drains: the build finishes, the new generation takes over client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw client # [ 251.593969] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw. client # Running as unit: sw.service; invocation ID: ae3a99780f77415c8aaaca5bfe15a898 client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.08 seconds) client # [ 251.793795] nix[1936]: don't know how to build these paths: client # [ 251.794929] nix[1936]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker1 # [ 251.773119] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:38380 worker1 # [ 251.790793] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:24Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=0 worker1 # [ 251.797931] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:24Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 251.836122] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38380 worker1 # [ 251.856640] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:38380 client # [ 251.891183] nix[1936]: copying 1 paths to 'lb:50051'... worker1 # [ 251.871211] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38380 client # [ 251.894526] nix[1936]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'... worker1 # [ 251.886267] nix-daemon[832]: accepted connection from pid 1406, user nix-grpc-daemon (trusted) worker1 # [ 252.040187] nix-daemon[832]: accepted connection from pid 1564, user nix-grpc-daemon (trusted) worker1 # [ 252.058916] nix-daemon[832]: reaped child process 1567, status = succeeded lb # [ 252.031782] niks3-server[1111]: 2026/09/23 05:16:24 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 252.078863] nix-grpc-daemon[1515]: time=2026-09-23T05:16:24.584Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 252.080466] nix-grpc-daemon[1515]: time=2026-09-23T05:16:24.584Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)" lb # [ 252.066627] niks3-server[1111]: 2026/09/23 05:16:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign worker1 # [ 252.109078] nix-grpc-daemon[1515]: time=2026-09-23T05:16:24.615Z level=INFO msg="Uploading 1 narinfos" lb # [ 252.084169] niks3-server[1111]: 2026/09/23 05:16:24 INFO Signed narinfos id=54 count=1 lb # [ 252.086841] niks3-server[1111]: 2026/09/23 05:16:24 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst lb # [ 252.104610] niks3-server[1111]: 2026/09/23 05:16:24 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls lb # [ 252.108604] niks3-server[1111]: 2026/09/23 05:16:24 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete worker1 # [ 252.144510] nix-grpc-daemon[1515]: time=2026-09-23T05:16:24.651Z level=INFO msg="Upload complete. (239ms)" lb # [ 252.114743] niks3-server[1111]: 2026/09/23 05:16:24 INFO Completed upload id=54 lb # [ 252.116834] niks3-server[1111]: 2026/09/23 05:16:24 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo worker1 # [ 252.156146] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38380 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 252.171487] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38380 worker1 # [ 252.178008] nix-daemon[832]: accepted connection from pid 1406, user nix-grpc-daemon (trusted) worker1 # [ 252.182926] nix-daemon[832]: reaped child process 1562, status = succeeded worker1 # [ 252.194868] nix-daemon[832]: accepted connection from pid 1406, user nix-grpc-daemon (trusted) client # [ 252.213740] nix[1936]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051... worker1: must succeed: systemctl reload nix-grpc-daemon.service worker1 # [ 254.172336] systemd[1]: Reloading Nix worker-protocol over gRPC... worker1 # [ 254.208999] systemd[1]: Reloaded Nix worker-protocol over gRPC. worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.08 seconds) worker1: must succeed: pgrep -f 'read -t [9]0 x' worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2 worker2 # [ 254.561753] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:47332 client # don't know how to build these paths: client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker2 # [ 254.573505] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47332 worker1 # [ 254.527706] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:27Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=0 worker1 # [ 254.529774] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:27Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 254.644588] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 254.659309] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47332 client # copying 1 paths to 'lb:50051'... worker2 # [ 254.675138] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'... worker2 # [ 254.693299] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker1 # [ 254.755609] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:27Z level=info event=draining worker2 # [ 254.854824] nix-daemon[855]: accepted connection from pid 1872, user nix-grpc-daemon (trusted) worker2 # [ 254.874122] nix-daemon[855]: reaped child process 1875, status = succeeded lb # [ 254.778951] niks3-server[1111]: 2026/09/23 05:16:27 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 254.890804] nix-grpc-daemon[1805]: time=2026-09-23T05:16:27.343Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 254.894185] nix-grpc-daemon[1805]: time=2026-09-23T05:16:27.346Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)" lb # [ 254.813496] niks3-server[1111]: 2026/09/23 05:16:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign worker2 # [ 254.923645] nix-grpc-daemon[1805]: time=2026-09-23T05:16:27.376Z level=INFO msg="Uploading 1 narinfos" lb # [ 254.824886] niks3-server[1111]: 2026/09/23 05:16:27 INFO Signed narinfos id=55 count=1 lb # [ 254.836799] niks3-server[1111]: 2026/09/23 05:16:27 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst lb # [ 254.848660] niks3-server[1111]: 2026/09/23 05:16:27 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls lb # [ 254.863147] niks3-server[1111]: 2026/09/23 05:16:27 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete lb # [ 254.873225] niks3-server[1111]: 2026/09/23 05:16:27 INFO Completed upload id=55 lb # [ 254.875392] niks3-server[1111]: 2026/09/23 05:16:27 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo worker2 # [ 254.972310] nix-grpc-daemon[1805]: time=2026-09-23T05:16:27.425Z level=INFO msg="Upload complete. (261ms)" worker2 # [ 254.984575] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=0 paths=1 nar_bytes_in=528 worker2 # [ 255.000523] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 255.013987] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 255.018239] nix-daemon[855]: reaped child process 1870, status = succeeded client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051... worker2 # [ 255.032701] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) client # farm-a> worker2: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-during-drain worker2 # [ 256.106361] nix-daemon[855]: reaped child process 1879, status = succeeded worker2 # [ 256.218997] nix-daemon[855]: accepted connection from pid 1895, user nix-grpc-daemon (trusted) worker2 # [ 256.239316] nix-daemon[855]: reaped child process 1898, status = succeeded lb # [ 256.142436] niks3-server[1111]: 2026/09/23 05:16:28 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 256.255776] nix-grpc-daemon[1805]: time=2026-09-23T05:16:28.706Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 256.258954] nix-grpc-daemon[1805]: time=2026-09-23T05:16:28.706Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)" lb # [ 256.185491] niks3-server[1111]: 2026/09/23 05:16:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign worker2 # [ 256.295140] nix-grpc-daemon[1805]: time=2026-09-23T05:16:28.748Z level=INFO msg="Uploading 1 narinfos" lb # [ 256.199315] niks3-server[1111]: 2026/09/23 05:16:28 INFO Signed narinfos id=56 count=1 lb # [ 256.206986] niks3-server[1111]: 2026/09/23 05:16:28 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls lb # [ 256.214970] niks3-server[1111]: 2026/09/23 05:16:28 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker2 # [ 256.327623] nix-grpc-daemon[1805]: time=2026-09-23T05:16:28.780Z level=INFO msg="Upload complete. (278ms)" lb # [ 256.226867] niks3-server[1111]: 2026/09/23 05:16:28 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst lb # [ 256.234579] niks3-server[1111]: 2026/09/23 05:16:28 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete lb # [ 256.239299] niks3-server[1111]: 2026/09/23 05:16:28 INFO Completed upload id=56 worker1 # [ 256.276822] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:28Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=0 worker2 # [ 256.341226] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=1 worker1 # [ 256.278666] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:28Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.4:50051 clients=1 lb # [ 256.243953] niks3-server[1111]: 2026/09/23 05:16:28 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo worker2 # [ 256.359055] nix-daemon[855]: reaped child process 1877, status = succeeded worker2 # [ 256.401020] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 256.415382] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 256.429641] nix-grpc-daemon[1805]: time=2026-09-23T05:16:28.882Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'... worker2 # [ 256.440454] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 256.457172] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 256.614283] nix-daemon[855]: accepted connection from pid 1902, user nix-grpc-daemon (trusted) worker2 # [ 256.632983] nix-daemon[855]: reaped child process 1905, status = succeeded lb # [ 256.536559] niks3-server[1111]: 2026/09/23 05:16:29 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 256.647098] nix-grpc-daemon[1805]: time=2026-09-23T05:16:29.099Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 256.648776] nix-grpc-daemon[1805]: time=2026-09-23T05:16:29.100Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)" worker2 # [ 256.673613] nix-grpc-daemon[1805]: time=2026-09-23T05:16:29.126Z level=INFO msg="Uploading 1 narinfos" lb # [ 256.565456] niks3-server[1111]: 2026/09/23 05:16:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign lb # [ 256.579169] niks3-server[1111]: 2026/09/23 05:16:29 INFO Signed narinfos id=57 count=1 lb # [ 256.588699] niks3-server[1111]: 2026/09/23 05:16:29 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst lb # [ 256.602553] niks3-server[1111]: 2026/09/23 05:16:29 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls lb # [ 256.607310] niks3-server[1111]: 2026/09/23 05:16:29 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete worker2 # [ 256.717549] nix-grpc-daemon[1805]: time=2026-09-23T05:16:29.170Z level=INFO msg="Upload complete. (243ms)" lb # [ 256.616163] niks3-server[1111]: 2026/09/23 05:16:29 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo lb # [ 256.624076] niks3-server[1111]: 2026/09/23 05:16:29 INFO Completed upload id=57 worker2 # [ 256.728299] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=0 paths=1 nar_bytes_in=640 worker2 # [ 256.742522] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 256.752529] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 256.755968] nix-daemon[855]: reaped child process 1900, status = succeeded client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051... worker2 # [ 256.770452] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) client # farm-b> worker2: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-during-drain worker2 # [ 258.064405] nix-daemon[855]: reaped child process 1909, status = succeeded worker2 # [ 258.153959] nix-daemon[855]: accepted connection from pid 1925, user nix-grpc-daemon (trusted) worker2 # [ 258.171682] nix-daemon[855]: reaped child process 1928, status = succeeded lb # [ 258.077863] niks3-server[1111]: 2026/09/23 05:16:30 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 258.190294] nix-grpc-daemon[1805]: time=2026-09-23T05:16:30.642Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 258.191847] nix-grpc-daemon[1805]: time=2026-09-23T05:16:30.642Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)" lb # [ 258.116453] niks3-server[1111]: 2026/09/23 05:16:30 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls lb # [ 258.128186] niks3-server[1111]: 2026/09/23 05:16:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign worker2 # [ 258.234881] nix-grpc-daemon[1805]: time=2026-09-23T05:16:30.687Z level=INFO msg="Uploading 1 narinfos" lb # [ 258.140713] niks3-server[1111]: 2026/09/23 05:16:30 INFO Signed narinfos id=58 count=1 lb # [ 258.143077] niks3-server[1111]: 2026/09/23 05:16:30 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst lb # [ 258.153759] niks3-server[1111]: 2026/09/23 05:16:30 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv lb # [ 258.161986] niks3-server[1111]: 2026/09/23 05:16:30 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete worker2 # [ 258.263986] nix-grpc-daemon[1805]: time=2026-09-23T05:16:30.717Z level=INFO msg="Upload complete. (278ms)" lb # [ 258.165864] niks3-server[1111]: 2026/09/23 05:16:30 INFO Completed upload id=58 lb # [ 258.167697] niks3-server[1111]: 2026/09/23 05:16:30 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo worker1 # [ 258.208994] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:30Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=0 worker1 # [ 258.210896] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:30Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 258.277066] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=1 worker2 # [ 258.296248] nix-daemon[855]: reaped child process 1907, status = succeeded worker2 # [ 258.326069] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 258.341141] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 258.355500] nix-grpc-daemon[1805]: time=2026-09-23T05:16:30.808Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'... worker2 # [ 258.366672] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 258.380962] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 258.532709] nix-daemon[855]: accepted connection from pid 1932, user nix-grpc-daemon (trusted) worker2 # [ 258.548757] nix-daemon[855]: reaped child process 1935, status = succeeded lb # [ 258.454765] niks3-server[1111]: 2026/09/23 05:16:30 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 258.565437] nix-grpc-daemon[1805]: time=2026-09-23T05:16:31.018Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 258.566991] nix-grpc-daemon[1805]: time=2026-09-23T05:16:31.018Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)" lb # [ 258.484251] niks3-server[1111]: 2026/09/23 05:16:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign worker2 # [ 258.594729] nix-grpc-daemon[1805]: time=2026-09-23T05:16:31.047Z level=INFO msg="Uploading 1 narinfos" lb # [ 258.500614] niks3-server[1111]: 2026/09/23 05:16:31 INFO Signed narinfos id=59 count=1 lb # [ 258.503564] niks3-server[1111]: 2026/09/23 05:16:31 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst lb # [ 258.519761] niks3-server[1111]: 2026/09/23 05:16:31 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls lb # [ 258.525878] niks3-server[1111]: 2026/09/23 05:16:31 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete worker2 # [ 258.621604] nix-grpc-daemon[1805]: time=2026-09-23T05:16:31.074Z level=INFO msg="Upload complete. (223ms)" lb # [ 258.531677] niks3-server[1111]: 2026/09/23 05:16:31 INFO Completed upload id=59 worker2 # [ 258.632855] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=0 paths=1 nar_bytes_in=768 lb # [ 258.534252] niks3-server[1111]: 2026/09/23 05:16:31 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo worker2 # [ 258.642611] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 258.649106] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 258.653976] nix-daemon[855]: reaped child process 1930, status = succeeded client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051... client # farm-top> worker2: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker2 # [ 258.667294] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds lb # [ 259.199581] envoy[895]: conn peer=[2001:db8:1::1]:45824 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2987 tx=851 ms=3513 client # farm-top> LOG-farm-top-during-drain worker2 # [ 259.752565] nix-daemon[855]: reaped child process 1939, status = succeeded worker2 # [ 259.849631] nix-daemon[855]: accepted connection from pid 1955, user nix-grpc-daemon (trusted) worker2 # [ 259.867884] nix-daemon[855]: reaped child process 1958, status = succeeded lb # [ 259.773996] niks3-server[1111]: 2026/09/23 05:16:32 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 259.887101] nix-grpc-daemon[1805]: time=2026-09-23T05:16:32.338Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 259.889041] nix-grpc-daemon[1805]: time=2026-09-23T05:16:32.338Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)" lb # [ 259.816947] niks3-server[1111]: 2026/09/23 05:16:32 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls worker2 # [ 259.928742] nix-grpc-daemon[1805]: time=2026-09-23T05:16:32.381Z level=INFO msg="Uploading 1 narinfos" lb # [ 259.830541] niks3-server[1111]: 2026/09/23 05:16:32 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv lb # [ 259.839746] niks3-server[1111]: 2026/09/23 05:16:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign lb # [ 259.850134] niks3-server[1111]: 2026/09/23 05:16:32 INFO Signed narinfos id=60 count=1 worker2 # [ 259.952081] nix-grpc-daemon[1805]: time=2026-09-23T05:16:32.405Z level=INFO msg="Upload complete. (273ms)" lb # [ 259.853381] niks3-server[1111]: 2026/09/23 05:16:32 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst lb # [ 259.858070] niks3-server[1111]: 2026/09/23 05:16:32 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete lb # [ 259.862394] niks3-server[1111]: 2026/09/23 05:16:32 INFO Completed upload id=60 lb # [ 259.864503] niks3-server[1111]: 2026/09/23 05:16:32 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo worker2 # [ 259.965159] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=1 worker2 # [ 259.982396] nix-daemon[855]: reaped child process 1937, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 5.68 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.17 seconds) worker1: must succeed: pkill -f 'read -t [9]0 x' worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds) client: waiting for success: ! systemctl is-active sw worker1 # [ 260.227506] nix-daemon[832]: reaped child process 1571, status = succeeded worker1 # [ 260.322059] nix-daemon[832]: accepted connection from pid 1618, user nix-grpc-daemon (trusted) worker1 # [ 260.338593] nix-daemon[832]: reaped child process 1621, status = succeeded lb # [ 260.314176] niks3-server[1111]: 2026/09/23 05:16:32 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 260.356053] nix-grpc-daemon[1515]: time=2026-09-23T05:16:32.861Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 260.357517] nix-grpc-daemon[1515]: time=2026-09-23T05:16:32.861Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)" worker1 # [ 260.389697] nix-grpc-daemon[1515]: time=2026-09-23T05:16:32.896Z level=INFO msg="Uploading 1 narinfos" lb # [ 260.351905] niks3-server[1111]: 2026/09/23 05:16:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign lb # [ 260.367251] niks3-server[1111]: 2026/09/23 05:16:32 INFO Signed narinfos id=61 count=1 lb # [ 260.373541] niks3-server[1111]: 2026/09/23 05:16:32 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv lb # [ 260.387005] niks3-server[1111]: 2026/09/23 05:16:32 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls worker1 # [ 260.427255] nix-grpc-daemon[1515]: time=2026-09-23T05:16:32.933Z level=INFO msg="Upload complete. (213ms)" lb # [ 260.397132] niks3-server[1111]: 2026/09/23 05:16:32 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst lb # [ 260.404137] niks3-server[1111]: 2026/09/23 05:16:32 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete worker1 # [ 260.436750] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38380 duration_s=8 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=1 lb # [ 260.409465] niks3-server[1111]: 2026/09/23 05:16:32 INFO Completed upload id=61 lb # [ 260.412133] niks3-server[1111]: 2026/09/23 05:16:32 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo worker1 # [ 260.451616] nix-daemon[832]: reaped child process 1569, status = succeeded client # [ 260.502809] systemd[1]: sw.service: Deactivated successfully. client # [ 260.507856] systemd[1]: sw.service: Consumed 191ms CPU time over 8.909s wall clock time, 41.2M memory peak, 7.2K incoming IP traffic, 8.5K outgoing IP traffic. worker1 # [ 260.759063] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:33Z level=info event=signal_exit client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds) client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; } client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.05 seconds) worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service lb # [ 261.693284] niks3-server[1111]: 2026/09/23 05:16:34 INFO lead: released remote=192.168.1.3:40566 worker1 # [ 262.724934] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:35Z level=info event=scheduler_restarting worker2 # [ 262.796634] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:35Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 262.736799] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:35Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:38376 worker1 # [ 262.743044] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:35Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 262.747773] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:35Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 262.752454] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:35Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:38376 worker1 # [ 262.757643] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:35Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 262.781492] nix-daemon[832]: reaped child process 1481, status = succeeded worker2 # [ 262.996831] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:35Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 263.002217] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 262.955068] nix-grpc-daemon[1406]: ts=2026-09-23T05:16:35Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 263.203313] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:35Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 263.208625] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 263.409770] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:35Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 263.414850] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 263.615959] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 263.621058] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 263.681948] nix-grpc-daemon[1406]: E0000 00:00:1790140596.188220 1452 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true worker1 # [ 263.682281] nix-grpc-daemon[1406]: E0000 00:00:1790140596.188473 1452 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker2 # [ 263.823619] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 263.827795] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 263.848813] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 263.855333] systemd[1]: nix-grpc-daemon.service: Consumed 1.018s CPU time over 1min 58.300s wall clock time, 37.4M memory peak, 30.1K incoming IP traffic, 38.5K outgoing IP traffic. worker1 # [ 263.876801] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 264.024602] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 264.029616] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 264.020880] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:36Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 264.023631] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:36Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 264.027658] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:36Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 264.034788] nix-grpc-daemon[1635]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR lb # [ 264.013653] niks3-server[1111]: 2026/09/23 05:16:36 INFO lead: acquired remote=192.168.1.3:48532 worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.78 seconds) worker1 # [ 264.050377] nix-grpc-daemon[1635]: E0000 00:00:1790140596.539194 1635 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 264.080258] nix-grpc-daemon[1635]: E0000 00:00:1790140596.541294 1635 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 264.082352] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:36Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 264.083953] nix-grpc-daemon[1635]: E0000 00:00:1790140596.550928 1675 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 264.087681] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:36Z level=info event=scheduler_take_over worker1 # [ 264.089428] nix-grpc-daemon[1635]: E0000 00:00:1790140596.570092 1675 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 264.091840] nix-grpc-daemon[1635]: E0000 00:00:1790140596.570261 1675 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 264.094340] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 264.095136] nix-grpc-daemon[1635]: E0000 00:00:1790140596.593317 1675 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 264.103525] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 264.105405] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 264.230570] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 264.235928] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 264.306420] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 264.311522] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 264.437402] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 264.442474] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 264.512256] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 264.517337] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 264.643743] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 264.649212] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 264.718143] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 264.723259] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 264.851901] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 264.856184] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 264.923887] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 264.932142] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:37Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:55674 worker1 # [ 264.937680] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:37Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker2 # [ 265.052546] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 264.993713] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:37Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 264.998191] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:37Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:55674 worker1 # [ 265.003565] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:37Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.14 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 13.83 seconds) subtest: queue gauges show a burst while every build blocks lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) client: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]' client # [ 265.593545] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto --impure --expr "map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ \"q0\" \"q1\" \"q2\" \"q3\" ]". client # Running as unit: burst.service; invocation ID: 41f8e2843a464f08b12c6b713d1b987b client: (finished: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]', in 0.09 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1 # [ 265.776959] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:38Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 265.784826] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) client # [ 265.819954] nix[1992]: don't know how to build these paths: client # [ 265.821226] nix[1992]: /nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv client # [ 265.822575] nix[1992]: /nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv client # [ 265.824176] nix[1992]: /nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv client # [ 265.825459] nix[1992]: /nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker1 # [ 265.816235] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:38Z level=debug event=want client=1 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv cached=0 worker1 # [ 265.821360] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:38Z level=debug event=assigned drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 265.827528] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:38Z level=debug event=want client=1 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv cached=0 worker1 # [ 265.832569] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:38Z level=debug event=assigned drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv worker=192.168.1.4:50051 clients=1 worker1 # [ 265.837531] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:38Z level=debug event=want client=1 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv cached=0 worker1 # [ 265.842062] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:38Z level=debug event=want client=1 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv cached=0 worker1 # [ 265.869866] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker2 # [ 265.941323] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker1 # [ 265.887791] nix-grpc-daemon[1635]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 91 ms (attempt 1/5) worker2 # [ 265.959445] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47332 client # [ 265.924232] nix[1992]: copying 1 paths to 'lb:50051'... worker2 # [ 265.974558] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 client # [ 265.926599] nix[1992]: copying path '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' to 'lb:50051'... worker2 # [ 265.992150] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker1 # [ 265.987758] nix-grpc-daemon[1635]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 156 ms (attempt 2/5) worker2 # [ 266.150877] nix-daemon[855]: accepted connection from pid 1962, user nix-grpc-daemon (trusted) worker2 # [ 266.173111] nix-daemon[855]: reaped child process 1965, status = succeeded lb # [ 266.071395] niks3-server[1111]: 2026/09/23 05:16:38 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 266.183365] nix-grpc-daemon[1805]: time=2026-09-23T05:16:38.635Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 266.185174] nix-grpc-daemon[1805]: time=2026-09-23T05:16:38.635Z level=INFO msg="Uploading dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv (416B)" lb # [ 266.108830] niks3-server[1111]: 2026/09/23 05:16:38 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.ls worker1 # [ 266.145825] nix-grpc-daemon[1635]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 305 ms (attempt 3/5) worker2 # [ 266.222052] nix-grpc-daemon[1805]: time=2026-09-23T05:16:38.674Z level=INFO msg="Uploading 1 narinfos" lb # [ 266.120893] niks3-server[1111]: 2026/09/23 05:16:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign lb # [ 266.131835] niks3-server[1111]: 2026/09/23 05:16:38 INFO Signed narinfos id=62 count=1 lb # [ 266.140294] niks3-server[1111]: 2026/09/23 05:16:38 INFO Registered completed upload object_key=nar/06mdwsnf16qmfi86y8r33asi4vcrp5z7kdxqki87psgfpgvvpdki.nar.zst lb # [ 266.151634] niks3-server[1111]: 2026/09/23 05:16:38 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete lb # [ 266.157639] niks3-server[1111]: 2026/09/23 05:16:38 INFO Completed upload id=62 worker2 # [ 266.259538] nix-grpc-daemon[1805]: time=2026-09-23T05:16:38.712Z level=INFO msg="Upload complete. (250ms)" lb # [ 266.162351] niks3-server[1111]: 2026/09/23 05:16:38 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.narinfo worker2 # [ 266.271660] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=0 paths=1 nar_bytes_in=416 worker2 # [ 266.287208] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 266.294779] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 266.300269] nix-daemon[855]: reaped child process 1960, status = succeeded client # [ 266.261829] nix[1992]: building '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' on lb:50051... worker2 # [ 266.315793] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker1 # [ 266.452902] nix-grpc-daemon[1635]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 683 ms (attempt 4/5) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker1 # [ 267.141433] nix-grpc-daemon[1635]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 267.145872] nix-grpc-daemon[1635]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 267.166268] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55690 client # [ 267.201127] nix[1992]: copying 1 paths to 'lb:50051'... worker1 # [ 267.181900] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 client # [ 267.205864] nix[1992]: copying path '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' to 'lb:50051'... worker1 # [ 267.197440] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 267.225439] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:39Z level=info event=niks3_push_started pid=1719 worker1 # [ 267.374728] nix-grpc-daemon[1719]: time=2026-09-23T05:16:39.880Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 267.477356] nix-daemon[832]: accepted connection from pid 1731, user nix-grpc-daemon (trusted) worker1 # [ 267.496147] nix-daemon[832]: reaped child process 1734, status = succeeded lb # [ 267.475037] niks3-server[1111]: 2026/09/23 05:16:39 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 267.517736] nix-grpc-daemon[1719]: time=2026-09-23T05:16:40.023Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 267.519234] nix-grpc-daemon[1719]: time=2026-09-23T05:16:40.023Z level=INFO msg="Uploading d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv (416B)" lb # [ 267.524823] niks3-server[1111]: 2026/09/23 05:16:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign worker1 # [ 267.562831] nix-grpc-daemon[1719]: time=2026-09-23T05:16:40.068Z level=INFO msg="Uploading 1 narinfos" lb # [ 267.531789] niks3-server[1111]: 2026/09/23 05:16:40 INFO Signed narinfos id=63 count=1 lb # [ 267.539846] niks3-server[1111]: 2026/09/23 05:16:40 INFO Registered completed upload object_key=nar/06ismykzsix0w29371dm1hbzzf3ljf9jn7dyh8skp5wqfkkw0vlk.nar.zst lb # [ 267.550166] niks3-server[1111]: 2026/09/23 05:16:40 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.ls lb # [ 267.559044] niks3-server[1111]: 2026/09/23 05:16:40 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete lb # [ 267.566702] niks3-server[1111]: 2026/09/23 05:16:40 INFO Completed upload id=63 worker1 # [ 267.597767] nix-grpc-daemon[1719]: time=2026-09-23T05:16:40.104Z level=INFO msg="Upload complete. (220ms)" lb # [ 267.572252] niks3-server[1111]: 2026/09/23 05:16:40 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.narinfo worker1 # [ 267.608251] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 267.623411] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 267.630289] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 267.638146] nix-daemon[832]: reaped child process 1717, status = succeeded worker1 # [ 267.646662] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) client # [ 267.666178] nix[1992]: building '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' on lb:50051... worker2 # [ 268.586006] nix-daemon[855]: reaped child process 1969, status = succeeded worker2 # [ 268.701761] nix-daemon[855]: accepted connection from pid 1994, user nix-grpc-daemon (trusted) worker2 # [ 268.722635] nix-daemon[855]: reaped child process 1997, status = succeeded lb # [ 268.626748] niks3-server[1111]: 2026/09/23 05:16:41 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 268.740438] nix-grpc-daemon[1805]: time=2026-09-23T05:16:41.191Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 268.742139] nix-grpc-daemon[1805]: time=2026-09-23T05:16:41.191Z level=INFO msg="Uploading y3qkgrgq2zzpgvi2rf5chq94i627wkdb-slow-q3 (120B)" lb # [ 268.671158] niks3-server[1111]: 2026/09/23 05:16:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign worker2 # [ 268.782201] nix-grpc-daemon[1805]: time=2026-09-23T05:16:41.235Z level=INFO msg="Uploading 1 narinfos" lb # [ 268.685008] niks3-server[1111]: 2026/09/23 05:16:41 INFO Signed narinfos id=64 count=1 lb # [ 268.697222] niks3-server[1111]: 2026/09/23 05:16:41 INFO Registered completed upload object_key=log/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv lb # [ 268.708674] niks3-server[1111]: 2026/09/23 05:16:41 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.ls lb # [ 268.723908] niks3-server[1111]: 2026/09/23 05:16:41 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete lb # [ 268.737232] niks3-server[1111]: 2026/09/23 05:16:41 INFO Registered completed upload object_key=nar/1v1grzkvw3yj09x0dkdm2avihvqvnri156qzp768hk85zbw7lkn9.nar.zst lb # [ 268.743571] niks3-server[1111]: 2026/09/23 05:16:41 INFO Completed upload id=64 worker2 # [ 268.845169] nix-grpc-daemon[1805]: time=2026-09-23T05:16:41.298Z level=INFO msg="Upload complete. (316ms)" lb # [ 268.747021] niks3-server[1111]: 2026/09/23 05:16:41 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.narinfo worker2 # [ 268.857202] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=2 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv assign_id=2 outputs=1 worker1 # [ 268.790117] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:41Z level=debug event=assigned drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 268.879462] nix-daemon[855]: reaped child process 1967, status = succeeded worker2 # [ 268.906279] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 268.923318] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47332 client # [ 268.889232] nix[1992]: copying 1 paths to 'lb:50051'... worker2 # [ 268.938082] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 client # [ 268.889554] nix[1992]: copying path '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' to 'lb:50051'... worker2 # [ 268.955351] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 269.117177] nix-daemon[855]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) worker2 # [ 269.138460] nix-daemon[855]: reaped child process 2004, status = succeeded lb # [ 269.041240] niks3-server[1111]: 2026/09/23 05:16:41 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 269.151819] nix-grpc-daemon[1805]: time=2026-09-23T05:16:41.604Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 269.153573] nix-grpc-daemon[1805]: time=2026-09-23T05:16:41.604Z level=INFO msg="Uploading jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv (416B)" lb # [ 269.072555] niks3-server[1111]: 2026/09/23 05:16:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign worker2 # [ 269.180577] nix-grpc-daemon[1805]: time=2026-09-23T05:16:41.633Z level=INFO msg="Uploading 1 narinfos" lb # [ 269.086317] niks3-server[1111]: 2026/09/23 05:16:41 INFO Signed narinfos id=65 count=1 lb # [ 269.094359] niks3-server[1111]: 2026/09/23 05:16:41 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.ls worker2 # [ 269.208793] nix-grpc-daemon[1805]: time=2026-09-23T05:16:41.661Z level=INFO msg="Upload complete. (236ms)" lb # [ 269.107625] niks3-server[1111]: 2026/09/23 05:16:41 INFO Registered completed upload object_key=nar/1fyfjzlx8vymsfmdv5cv8ymgjkr18fwm737qvys29plwwiwmyvwr.nar.zst lb # [ 269.114858] niks3-server[1111]: 2026/09/23 05:16:41 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete worker2 # [ 269.216415] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=0 paths=1 nar_bytes_in=416 lb # [ 269.119135] niks3-server[1111]: 2026/09/23 05:16:41 INFO Completed upload id=65 lb # [ 269.121323] niks3-server[1111]: 2026/09/23 05:16:41 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.narinfo worker2 # [ 269.228488] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 269.239507] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 269.244825] nix-daemon[855]: reaped child process 1999, status = succeeded client # [ 269.204696] nix[1992]: building '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' on lb:50051... worker2 # [ 269.257278] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) lb # [ 269.197993] envoy[895]: conn peer=[2001:db8:1::1]:47282 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9582 tx=4429 ms=5421 lb # [ 269.202998] envoy[895]: conn peer=[2001:db8:1::1]:47276 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2978 tx=1516 ms=8690 lb # [ 269.208004] envoy[895]: conn peer=[2001:db8:1::3]:39458 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2388 tx=995 ms=117177 worker1 # [ 269.489373] nix-daemon[832]: reaped child process 1739, status = succeeded worker1 # [ 269.599478] nix-daemon[832]: accepted connection from pid 1764, user nix-grpc-daemon (trusted) worker1 # [ 269.618007] nix-daemon[832]: reaped child process 1767, status = succeeded lb # [ 269.593916] niks3-server[1111]: 2026/09/23 05:16:42 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 269.636634] nix-grpc-daemon[1719]: time=2026-09-23T05:16:42.141Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 269.641446] nix-grpc-daemon[1719]: time=2026-09-23T05:16:42.141Z level=INFO msg="Uploading k6zivhvzgmlyxfmy64fzd759bxlhf9b5-slow-q0 (120B)" lb # [ 269.641995] niks3-server[1111]: 2026/09/23 05:16:42 INFO Registered completed upload object_key=nar/0xr809jj32p5n6mnc6i7lqsz48wrm5f9jp14m77k724041711iq7.nar.zst worker1 # [ 269.689359] nix-grpc-daemon[1719]: time=2026-09-23T05:16:42.195Z level=INFO msg="Uploading 1 narinfos" lb # [ 269.657207] niks3-server[1111]: 2026/09/23 05:16:42 INFO Registered completed upload object_key=log/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv lb # [ 269.667956] niks3-server[1111]: 2026/09/23 05:16:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign lb # [ 269.675788] niks3-server[1111]: 2026/09/23 05:16:42 INFO Signed narinfos id=66 count=1 worker1 # [ 269.716917] nix-grpc-daemon[1719]: time=2026-09-23T05:16:42.223Z level=INFO msg="Upload complete. (258ms)" lb # [ 269.686640] niks3-server[1111]: 2026/09/23 05:16:42 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.ls lb # [ 269.693213] niks3-server[1111]: 2026/09/23 05:16:42 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete lb # [ 269.699694] niks3-server[1111]: 2026/09/23 05:16:42 INFO Completed upload id=66 worker1 # [ 269.728356] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=2 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv assign_id=1 outputs=1 lb # [ 269.702579] niks3-server[1111]: 2026/09/23 05:16:42 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.narinfo worker1 # [ 269.747230] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:42Z level=debug event=assigned drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 269.752490] nix-daemon[832]: reaped child process 1737, status = succeeded worker1 # [ 269.773560] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 269.793453] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55690 client # [ 269.828256] nix[1992]: copying 1 paths to 'lb:50051'... worker1 # [ 269.810259] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 client # [ 269.832372] nix[1992]: copying path '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' to 'lb:50051'... worker1 # [ 269.825735] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 269.974773] nix-daemon[832]: accepted connection from pid 1771, user nix-grpc-daemon (trusted) worker1 # [ 269.997401] nix-daemon[832]: reaped child process 1774, status = succeeded lb # [ 269.964281] niks3-server[1111]: 2026/09/23 05:16:42 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 270.005639] nix-grpc-daemon[1719]: time=2026-09-23T05:16:42.512Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 270.007161] nix-grpc-daemon[1719]: time=2026-09-23T05:16:42.512Z level=INFO msg="Uploading pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv (416B)" lb # [ 269.997751] niks3-server[1111]: 2026/09/23 05:16:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign worker1 # [ 270.038075] nix-grpc-daemon[1719]: time=2026-09-23T05:16:42.544Z level=INFO msg="Uploading 1 narinfos" lb # [ 270.012974] niks3-server[1111]: 2026/09/23 05:16:42 INFO Signed narinfos id=67 count=1 lb # [ 270.020760] niks3-server[1111]: 2026/09/23 05:16:42 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.ls lb # [ 270.032226] niks3-server[1111]: 2026/09/23 05:16:42 INFO Registered completed upload object_key=nar/1k21jrz411xc9gkca2pnr9mrjq1s3bg3zhsam3nnv4j066dgzgm7.nar.zst worker1 # [ 270.068900] nix-grpc-daemon[1719]: time=2026-09-23T05:16:42.575Z level=INFO msg="Upload complete. (227ms)" lb # [ 270.037683] niks3-server[1111]: 2026/09/23 05:16:42 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete lb # [ 270.041426] niks3-server[1111]: 2026/09/23 05:16:42 INFO Completed upload id=67 lb # [ 270.048243] niks3-server[1111]: 2026/09/23 05:16:42 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.narinfo worker1 # [ 270.079210] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 270.092373] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 270.100338] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 270.102877] nix-daemon[832]: reaped child process 1769, status = succeeded worker1 # [ 270.114063] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) client # [ 270.132922] nix[1992]: building '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' on lb:50051... worker2 # [ 270.936283] nix-daemon[855]: reaped child process 2008, status = succeeded worker2 # [ 271.034284] nix-daemon[855]: accepted connection from pid 2033, user nix-grpc-daemon (trusted) worker2 # [ 271.055098] nix-daemon[855]: reaped child process 2036, status = succeeded lb # [ 270.958306] niks3-server[1111]: 2026/09/23 05:16:43 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 271.072408] nix-grpc-daemon[1805]: time=2026-09-23T05:16:43.522Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 271.074008] nix-grpc-daemon[1805]: time=2026-09-23T05:16:43.522Z level=INFO msg="Uploading gjyi8j90pk5flrsqqdhj6h132pcb7n33-slow-q2 (120B)" lb # [ 271.001099] niks3-server[1111]: 2026/09/23 05:16:43 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.ls lb # [ 271.015400] niks3-server[1111]: 2026/09/23 05:16:43 INFO Registered completed upload object_key=log/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv lb # [ 271.026164] niks3-server[1111]: 2026/09/23 05:16:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign worker2 # [ 271.135065] nix-grpc-daemon[1805]: time=2026-09-23T05:16:43.587Z level=INFO msg="Uploading 1 narinfos" lb # [ 271.036690] niks3-server[1111]: 2026/09/23 05:16:43 INFO Signed narinfos id=68 count=1 lb # [ 271.042807] niks3-server[1111]: 2026/09/23 05:16:43 INFO Registered completed upload object_key=nar/15imdx7zyy6mzjjv6qvdzywh0d9w1n6ikc0rq4rq95aw9x6swfzj.nar.zst lb # [ 271.055553] niks3-server[1111]: 2026/09/23 05:16:43 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete lb # [ 271.063428] niks3-server[1111]: 2026/09/23 05:16:43 INFO Completed upload id=68 worker2 # [ 271.164151] nix-grpc-daemon[1805]: time=2026-09-23T05:16:43.617Z level=INFO msg="Upload complete. (296ms)" lb # [ 271.066239] niks3-server[1111]: 2026/09/23 05:16:43 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.narinfo worker2 # [ 271.177268] nix-grpc-daemon[1723]: ts=2026-09-23T05:16:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=1 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv assign_id=3 outputs=1 worker2 # [ 271.195230] nix-daemon[855]: reaped child process 2006, status = succeeded worker1 # [ 271.992114] nix-daemon[832]: reaped child process 1778, status = succeeded worker1 # [ 272.093857] nix-daemon[832]: accepted connection from pid 1803, user nix-grpc-daemon (trusted) worker1 # [ 272.110366] nix-daemon[832]: reaped child process 1806, status = succeeded lb # [ 272.084619] niks3-server[1111]: 2026/09/23 05:16:44 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 272.126576] nix-grpc-daemon[1719]: time=2026-09-23T05:16:44.632Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 272.128368] nix-grpc-daemon[1719]: time=2026-09-23T05:16:44.632Z level=INFO msg="Uploading l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx-slow-q1 (120B)" lb # [ 272.126455] niks3-server[1111]: 2026/09/23 05:16:44 INFO Registered completed upload object_key=nar/0c1c367r3msa4fz6imwp70zxi4fxvrkb86v3hv921wsrf13yd8v3.nar.zst worker1 # [ 272.169496] nix-grpc-daemon[1719]: time=2026-09-23T05:16:44.675Z level=INFO msg="Uploading 1 narinfos" lb # [ 272.141880] niks3-server[1111]: 2026/09/23 05:16:44 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.ls lb # [ 272.151170] niks3-server[1111]: 2026/09/23 05:16:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign lb # [ 272.161766] niks3-server[1111]: 2026/09/23 05:16:44 INFO Signed narinfos id=69 count=1 worker1 # [ 272.192727] nix-grpc-daemon[1719]: time=2026-09-23T05:16:44.699Z level=INFO msg="Upload complete. (264ms)" lb # [ 272.163960] niks3-server[1111]: 2026/09/23 05:16:44 INFO Registered completed upload object_key=log/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv lb # [ 272.170274] niks3-server[1111]: 2026/09/23 05:16:44 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete lb # [ 272.174525] niks3-server[1111]: 2026/09/23 05:16:44 INFO Completed upload id=69 lb # [ 272.177119] niks3-server[1111]: 2026/09/23 05:16:44 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.narinfo worker1 # [ 272.205107] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=2 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv assign_id=4 outputs=1 worker1 # [ 272.224328] nix-daemon[832]: reaped child process 1776, status = succeeded client # [ 272.273325] systemd[1]: burst.service: Deactivated successfully. client # [ 272.277437] systemd[1]: burst.service: Consumed 226ms CPU time over 6.683s wall clock time, 53.7M memory peak, 13.8K incoming IP traffic, 18.5K outgoing IP traffic. client: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; } client: (finished: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: queue gauges show a burst while every build blocks, in 8.03 seconds) subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1 worker1 # [ 273.589317] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 273.596097] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 273.641159] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:46Z level=debug event=want client=2 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=0 worker1 # [ 273.648251] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:46Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 273.654171] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:46Z level=debug event=want client=2 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=0 worker1 # [ 273.659443] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:46Z level=debug event=want client=2 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=0 worker1 # [ 273.688496] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 273.704515] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 273.718791] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 273.733723] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 273.885578] nix-daemon[832]: accepted connection from pid 1819, user nix-grpc-daemon (trusted) worker1 # [ 273.907802] nix-daemon[832]: reaped child process 1822, status = succeeded lb # [ 273.877459] niks3-server[1111]: 2026/09/23 05:16:46 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 273.918595] nix-grpc-daemon[1719]: time=2026-09-23T05:16:46.424Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 273.920312] nix-grpc-daemon[1719]: time=2026-09-23T05:16:46.424Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)" lb # [ 273.908263] niks3-server[1111]: 2026/09/23 05:16:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign worker1 # [ 273.947387] nix-grpc-daemon[1719]: time=2026-09-23T05:16:46.453Z level=INFO msg="Uploading 1 narinfos" lb # [ 273.921682] niks3-server[1111]: 2026/09/23 05:16:46 INFO Signed narinfos id=70 count=1 lb # [ 273.929884] niks3-server[1111]: 2026/09/23 05:16:46 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst worker1 # [ 273.976488] nix-grpc-daemon[1719]: time=2026-09-23T05:16:46.483Z level=INFO msg="Upload complete. (226ms)" lb # [ 273.944784] niks3-server[1111]: 2026/09/23 05:16:46 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls lb # [ 273.955670] niks3-server[1111]: 2026/09/23 05:16:46 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete worker1 # [ 273.986347] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 273.963245] niks3-server[1111]: 2026/09/23 05:16:46 INFO Completed upload id=70 lb # [ 273.965326] niks3-server[1111]: 2026/09/23 05:16:46 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo worker1 # [ 273.999440] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 274.008304] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 274.011100] nix-daemon[832]: reaped child process 1817, status = succeeded worker1 # [ 274.021701] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 274.689971] nix-daemon[832]: reaped child process 1826, status = succeeded worker1 # [ 274.821187] nix-daemon[832]: accepted connection from pid 1842, user nix-grpc-daemon (trusted) worker1 # [ 274.837802] nix-daemon[832]: reaped child process 1845, status = succeeded lb # [ 274.812277] niks3-server[1111]: 2026/09/23 05:16:47 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 274.854400] nix-grpc-daemon[1719]: time=2026-09-23T05:16:47.359Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 274.856200] nix-grpc-daemon[1719]: time=2026-09-23T05:16:47.359Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)" lb # [ 274.840567] niks3-server[1111]: 2026/09/23 05:16:47 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv lb # [ 274.862886] niks3-server[1111]: 2026/09/23 05:16:47 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst worker1 # [ 274.909440] nix-grpc-daemon[1719]: time=2026-09-23T05:16:47.415Z level=INFO msg="Uploading 1 narinfos" lb # [ 274.872113] niks3-server[1111]: 2026/09/23 05:16:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign lb # [ 274.882934] niks3-server[1111]: 2026/09/23 05:16:47 INFO Signed narinfos id=71 count=1 lb # [ 274.889843] niks3-server[1111]: 2026/09/23 05:16:47 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls lb # [ 274.902488] niks3-server[1111]: 2026/09/23 05:16:47 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete lb # [ 274.910619] niks3-server[1111]: 2026/09/23 05:16:47 INFO Completed upload id=71 worker1 # [ 274.941912] nix-grpc-daemon[1719]: time=2026-09-23T05:16:47.447Z level=INFO msg="Upload complete. (304ms)" lb # [ 274.914187] niks3-server[1111]: 2026/09/23 05:16:47 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo worker1 # [ 274.960969] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=0 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=5 outputs=1 worker1 # [ 274.980315] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:47Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 274.988105] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 274.998026] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:47Z level=debug event=want client=2 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=0 worker1 # [ 275.007585] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 275.014233] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 275.022446] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 275.026310] nix-daemon[832]: reaped child process 1824, status = succeeded worker1 # [ 275.141507] nix-daemon[832]: accepted connection from pid 1851, user nix-grpc-daemon (trusted) worker1 # [ 275.158678] nix-daemon[832]: reaped child process 1854, status = succeeded lb # [ 275.133211] niks3-server[1111]: 2026/09/23 05:16:47 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 275.174133] nix-grpc-daemon[1719]: time=2026-09-23T05:16:47.680Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 275.175582] nix-grpc-daemon[1719]: time=2026-09-23T05:16:47.680Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)" lb # [ 275.164245] niks3-server[1111]: 2026/09/23 05:16:47 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls worker1 # [ 275.206635] nix-grpc-daemon[1719]: time=2026-09-23T05:16:47.713Z level=INFO msg="Uploading 1 narinfos" lb # [ 275.177892] niks3-server[1111]: 2026/09/23 05:16:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign lb # [ 275.188253] niks3-server[1111]: 2026/09/23 05:16:47 INFO Signed narinfos id=72 count=1 lb # [ 275.190731] niks3-server[1111]: 2026/09/23 05:16:47 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst worker1 # [ 275.235794] nix-grpc-daemon[1719]: time=2026-09-23T05:16:47.742Z level=INFO msg="Upload complete. (222ms)" lb # [ 275.204759] niks3-server[1111]: 2026/09/23 05:16:47 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete lb # [ 275.210332] niks3-server[1111]: 2026/09/23 05:16:47 INFO Completed upload id=72 worker1 # [ 275.242273] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 275.213906] niks3-server[1111]: 2026/09/23 05:16:47 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo worker1 # [ 275.252479] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 275.261832] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 275.266675] nix-daemon[832]: reaped child process 1849, status = succeeded worker1 # [ 275.278971] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 276.316782] nix-daemon[832]: reaped child process 1858, status = succeeded worker1 # [ 276.430931] nix-daemon[832]: accepted connection from pid 1874, user nix-grpc-daemon (trusted) worker1 # [ 276.449337] nix-daemon[832]: reaped child process 1877, status = succeeded lb # [ 276.416710] niks3-server[1111]: 2026/09/23 05:16:48 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 276.460374] nix-grpc-daemon[1719]: time=2026-09-23T05:16:48.965Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 276.461825] nix-grpc-daemon[1719]: time=2026-09-23T05:16:48.965Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)" lb # [ 276.467815] niks3-server[1111]: 2026/09/23 05:16:48 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls worker1 # [ 276.507886] nix-grpc-daemon[1719]: time=2026-09-23T05:16:49.014Z level=INFO msg="Uploading 1 narinfos" lb # [ 276.477439] niks3-server[1111]: 2026/09/23 05:16:48 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst lb # [ 276.488541] niks3-server[1111]: 2026/09/23 05:16:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign lb # [ 276.491644] niks3-server[1111]: 2026/09/23 05:16:48 INFO Signed narinfos id=73 count=1 lb # [ 276.496279] niks3-server[1111]: 2026/09/23 05:16:48 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv lb # [ 276.503502] niks3-server[1111]: 2026/09/23 05:16:49 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete lb # [ 276.509134] niks3-server[1111]: 2026/09/23 05:16:49 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo lb # [ 276.513758] niks3-server[1111]: 2026/09/23 05:16:49 INFO Completed upload id=73 worker1 # [ 276.545301] nix-grpc-daemon[1719]: time=2026-09-23T05:16:49.051Z level=INFO msg="Upload complete. (257ms)" worker1 # [ 276.565182] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=6 outputs=1 worker1 # [ 276.581203] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:49Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 276.590967] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 276.596869] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:49Z level=debug event=want client=2 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=0 worker1 # [ 276.608294] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 276.611909] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 276.623877] nix-daemon[832]: reaped child process 1856, status = succeeded worker1 # [ 276.628373] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 276.748836] nix-daemon[832]: accepted connection from pid 1883, user nix-grpc-daemon (trusted) worker1 # [ 276.766866] nix-daemon[832]: reaped child process 1886, status = succeeded lb # [ 276.737705] niks3-server[1111]: 2026/09/23 05:16:49 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 276.778592] nix-grpc-daemon[1719]: time=2026-09-23T05:16:49.284Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 276.780101] nix-grpc-daemon[1719]: time=2026-09-23T05:16:49.284Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)" worker1 # [ 276.798456] nix-grpc-daemon[1719]: time=2026-09-23T05:16:49.304Z level=INFO msg="Uploading 1 narinfos" lb # [ 276.766197] niks3-server[1111]: 2026/09/23 05:16:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign lb # [ 276.778107] niks3-server[1111]: 2026/09/23 05:16:49 INFO Signed narinfos id=74 count=1 lb # [ 276.788476] niks3-server[1111]: 2026/09/23 05:16:49 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls lb # [ 276.800727] niks3-server[1111]: 2026/09/23 05:16:49 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst worker1 # [ 276.839719] nix-grpc-daemon[1719]: time=2026-09-23T05:16:49.346Z level=INFO msg="Upload complete. (216ms)" lb # [ 276.808633] niks3-server[1111]: 2026/09/23 05:16:49 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete lb # [ 276.817331] niks3-server[1111]: 2026/09/23 05:16:49 INFO Completed upload id=74 worker1 # [ 276.848239] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 276.819688] niks3-server[1111]: 2026/09/23 05:16:49 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo worker1 # [ 276.862314] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 276.870817] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 276.874265] nix-daemon[832]: reaped child process 1881, status = succeeded worker1 # [ 276.884123] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 278.340795] nix-daemon[832]: reaped child process 1890, status = succeeded worker1 # [ 278.457188] nix-daemon[832]: accepted connection from pid 1906, user nix-grpc-daemon (trusted) worker1 # [ 278.473372] nix-daemon[832]: reaped child process 1909, status = succeeded lb # [ 278.448002] niks3-server[1111]: 2026/09/23 05:16:50 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 278.491595] nix-grpc-daemon[1719]: time=2026-09-23T05:16:50.996Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 278.493739] nix-grpc-daemon[1719]: time=2026-09-23T05:16:50.996Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)" lb # [ 278.487799] niks3-server[1111]: 2026/09/23 05:16:50 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker1 # [ 278.532041] nix-grpc-daemon[1719]: time=2026-09-23T05:16:51.038Z level=INFO msg="Uploading 1 narinfos" lb # [ 278.500898] niks3-server[1111]: 2026/09/23 05:16:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign lb # [ 278.512327] niks3-server[1111]: 2026/09/23 05:16:51 INFO Signed narinfos id=75 count=1 lb # [ 278.518020] niks3-server[1111]: 2026/09/23 05:16:51 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls lb # [ 278.528281] niks3-server[1111]: 2026/09/23 05:16:51 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst worker1 # [ 278.567368] nix-grpc-daemon[1719]: time=2026-09-23T05:16:51.073Z level=INFO msg="Upload complete. (268ms)" lb # [ 278.536847] niks3-server[1111]: 2026/09/23 05:16:51 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete lb # [ 278.540699] niks3-server[1111]: 2026/09/23 05:16:51 INFO Completed upload id=75 lb # [ 278.544490] niks3-server[1111]: 2026/09/23 05:16:51 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo worker1 # [ 278.585462] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=7 outputs=1 worker1 # [ 278.604725] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:51Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 278.613714] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 278.619360] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:51Z level=debug event=want client=2 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=0 worker1 # [ 278.625179] nix-daemon[832]: reaped child process 1888, status = succeeded worker1 # [ 278.627236] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 278.632759] nix-grpc-daemon[1719]: time=2026-09-23T05:16:51.138Z level=INFO msg="All 1 paths already cached" worker1 # [ 278.636797] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 278.647506] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 278.779886] nix-daemon[832]: accepted connection from pid 1915, user nix-grpc-daemon (trusted) worker1 # [ 278.798713] nix-daemon[832]: reaped child process 1918, status = succeeded lb # [ 278.770546] niks3-server[1111]: 2026/09/23 05:16:51 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 278.811901] nix-grpc-daemon[1719]: time=2026-09-23T05:16:51.318Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 278.813492] nix-grpc-daemon[1719]: time=2026-09-23T05:16:51.318Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)" worker1 # [ 278.831940] nix-grpc-daemon[1719]: time=2026-09-23T05:16:51.338Z level=INFO msg="Uploading 1 narinfos" lb # [ 278.799665] niks3-server[1111]: 2026/09/23 05:16:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign lb # [ 278.814376] niks3-server[1111]: 2026/09/23 05:16:51 INFO Signed narinfos id=76 count=1 lb # [ 278.823219] niks3-server[1111]: 2026/09/23 05:16:51 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst lb # [ 278.833196] niks3-server[1111]: 2026/09/23 05:16:51 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls worker1 # [ 278.875024] nix-grpc-daemon[1719]: time=2026-09-23T05:16:51.381Z level=INFO msg="Upload complete. (216ms)" lb # [ 278.843515] niks3-server[1111]: 2026/09/23 05:16:51 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete lb # [ 278.852472] niks3-server[1111]: 2026/09/23 05:16:51 INFO Completed upload id=76 worker1 # [ 278.883737] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 278.855991] niks3-server[1111]: 2026/09/23 05:16:51 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo worker1 # [ 278.895977] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 278.905513] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 278.908261] nix-daemon[832]: reaped child process 1913, status = succeeded worker1 # [ 278.918038] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) lb # [ 279.201287] envoy[895]: conn peer=[2001:db8:1::1]:48088 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9675 tx=5207 ms=6466 worker1 # [ 279.953889] nix-daemon[832]: reaped child process 1922, status = succeeded worker1 # [ 280.060190] nix-daemon[832]: accepted connection from pid 1938, user nix-grpc-daemon (trusted) worker1 # [ 280.079769] nix-daemon[832]: reaped child process 1941, status = succeeded lb # [ 280.054333] niks3-server[1111]: 2026/09/23 05:16:52 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 280.097300] nix-grpc-daemon[1719]: time=2026-09-23T05:16:52.602Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 280.098878] nix-grpc-daemon[1719]: time=2026-09-23T05:16:52.602Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)" lb # [ 280.097831] niks3-server[1111]: 2026/09/23 05:16:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign worker1 # [ 280.139644] nix-grpc-daemon[1719]: time=2026-09-23T05:16:52.646Z level=INFO msg="Uploading 1 narinfos" lb # [ 280.114305] niks3-server[1111]: 2026/09/23 05:16:52 INFO Signed narinfos id=77 count=1 lb # [ 280.121579] niks3-server[1111]: 2026/09/23 05:16:52 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv lb # [ 280.132709] niks3-server[1111]: 2026/09/23 05:16:52 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls worker1 # [ 280.177441] nix-grpc-daemon[1719]: time=2026-09-23T05:16:52.684Z level=INFO msg="Upload complete. (248ms)" lb # [ 280.146357] niks3-server[1111]: 2026/09/23 05:16:52 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst lb # [ 280.153006] niks3-server[1111]: 2026/09/23 05:16:52 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete lb # [ 280.157385] niks3-server[1111]: 2026/09/23 05:16:52 INFO Completed upload id=77 worker1 # [ 280.188924] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=8 outputs=1 lb # [ 280.160548] niks3-server[1111]: 2026/09/23 05:16:52 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo worker1 # [ 280.203711] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:52Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 280.208320] nix-daemon[832]: reaped child process 1920, status = succeeded worker1 # [ 280.230958] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 280.238890] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:52Z level=debug event=want client=2 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=0 worker1 # [ 280.249751] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 280.263187] nix-grpc-daemon[1719]: time=2026-09-23T05:16:52.769Z level=INFO msg="All 1 paths already cached" worker1 # [ 280.271966] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 280.283488] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 280.427471] nix-daemon[832]: accepted connection from pid 1947, user nix-grpc-daemon (trusted) worker1 # [ 280.445206] nix-daemon[832]: reaped child process 1950, status = succeeded lb # [ 280.418726] niks3-server[1111]: 2026/09/23 05:16:52 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 280.459464] nix-grpc-daemon[1719]: time=2026-09-23T05:16:52.965Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 280.461126] nix-grpc-daemon[1719]: time=2026-09-23T05:16:52.965Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)" lb # [ 280.454349] niks3-server[1111]: 2026/09/23 05:16:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign worker1 # [ 280.493074] nix-grpc-daemon[1719]: time=2026-09-23T05:16:52.999Z level=INFO msg="Uploading 1 narinfos" lb # [ 280.466754] niks3-server[1111]: 2026/09/23 05:16:52 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls lb # [ 280.475938] niks3-server[1111]: 2026/09/23 05:16:52 INFO Signed narinfos id=78 count=1 worker1 # [ 280.516072] nix-grpc-daemon[1719]: time=2026-09-23T05:16:53.022Z level=INFO msg="Upload complete. (219ms)" lb # [ 280.485214] niks3-server[1111]: 2026/09/23 05:16:52 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst worker1 # [ 280.523727] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 280.493507] niks3-server[1111]: 2026/09/23 05:16:52 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete lb # [ 280.497103] niks3-server[1111]: 2026/09/23 05:16:52 INFO Completed upload id=78 lb # [ 280.499104] niks3-server[1111]: 2026/09/23 05:16:53 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo worker1 # [ 280.538800] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 280.545032] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 280.549890] nix-daemon[832]: reaped child process 1945, status = succeeded worker1 # [ 280.562734] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 282.273386] nix-daemon[832]: reaped child process 1954, status = succeeded worker1 # [ 282.459517] nix-daemon[832]: accepted connection from pid 1970, user nix-grpc-daemon (trusted) worker1 # [ 282.483750] nix-daemon[832]: reaped child process 1973, status = succeeded lb # [ 282.463495] niks3-server[1111]: 2026/09/23 05:16:54 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 282.512590] nix-grpc-daemon[1719]: time=2026-09-23T05:16:55.017Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 282.514227] nix-grpc-daemon[1719]: time=2026-09-23T05:16:55.017Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)" lb # [ 282.541808] niks3-server[1111]: 2026/09/23 05:16:55 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv lb # [ 282.557284] niks3-server[1111]: 2026/09/23 05:16:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign worker1 # [ 282.598199] nix-grpc-daemon[1719]: time=2026-09-23T05:16:55.103Z level=INFO msg="Uploading 1 narinfos" lb # [ 282.572333] niks3-server[1111]: 2026/09/23 05:16:55 INFO Signed narinfos id=79 count=1 lb # [ 282.578569] niks3-server[1111]: 2026/09/23 05:16:55 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst lb # [ 282.595270] niks3-server[1111]: 2026/09/23 05:16:55 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls lb # [ 282.605880] niks3-server[1111]: 2026/09/23 05:16:55 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete lb # [ 282.615150] niks3-server[1111]: 2026/09/23 05:16:55 INFO Completed upload id=79 lb # [ 282.618448] niks3-server[1111]: 2026/09/23 05:16:55 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo worker1 # [ 282.644191] nix-grpc-daemon[1719]: time=2026-09-23T05:16:55.150Z level=INFO msg="Upload complete. (467ms)" worker1 # [ 282.673475] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=2 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=9 outputs=1 worker1 # [ 282.696910] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:55Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 282.712436] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 282.721181] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:55Z level=debug event=want client=2 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=0 worker1 # [ 282.732729] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 282.741003] nix-daemon[832]: reaped child process 1952, status = succeeded worker1 # [ 282.745851] nix-grpc-daemon[1719]: time=2026-09-23T05:16:55.252Z level=INFO msg="All 1 paths already cached" worker1 # [ 282.759757] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 282.778804] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 283.024906] nix-daemon[832]: accepted connection from pid 1979, user nix-grpc-daemon (trusted) worker1 # [ 283.047442] nix-daemon[832]: reaped child process 1982, status = succeeded lb # [ 283.023895] niks3-server[1111]: 2026/09/23 05:16:55 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 283.069967] nix-grpc-daemon[1719]: time=2026-09-23T05:16:55.576Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 283.071527] nix-grpc-daemon[1719]: time=2026-09-23T05:16:55.576Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)" lb # [ 283.075672] niks3-server[1111]: 2026/09/23 05:16:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign worker1 # [ 283.120173] nix-grpc-daemon[1719]: time=2026-09-23T05:16:55.626Z level=INFO msg="Uploading 1 narinfos" lb # [ 283.094060] niks3-server[1111]: 2026/09/23 05:16:55 INFO Signed narinfos id=80 count=1 lb # [ 283.101097] niks3-server[1111]: 2026/09/23 05:16:55 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst lb # [ 283.115846] niks3-server[1111]: 2026/09/23 05:16:55 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls lb # [ 283.130397] niks3-server[1111]: 2026/09/23 05:16:55 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete worker1 # [ 283.171035] nix-grpc-daemon[1719]: time=2026-09-23T05:16:55.677Z level=INFO msg="Upload complete. (371ms)" lb # [ 283.142506] niks3-server[1111]: 2026/09/23 05:16:55 INFO Completed upload id=80 lb # [ 283.145618] niks3-server[1111]: 2026/09/23 05:16:55 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo worker1 # [ 283.184642] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 283.204434] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 283.219611] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 283.247311] nix-daemon[832]: reaped child process 1977, status = succeeded worker1 # [ 283.259297] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 284.960461] nix-daemon[832]: reaped child process 1986, status = succeeded worker1 # [ 285.061979] nix-daemon[832]: accepted connection from pid 2002, user nix-grpc-daemon (trusted) worker1 # [ 285.078908] nix-daemon[832]: reaped child process 2005, status = succeeded lb # [ 285.055402] niks3-server[1111]: 2026/09/23 05:16:57 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 285.099612] nix-grpc-daemon[1719]: time=2026-09-23T05:16:57.604Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 285.101649] nix-grpc-daemon[1719]: time=2026-09-23T05:16:57.604Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)" lb # [ 285.098503] niks3-server[1111]: 2026/09/23 05:16:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign worker1 # [ 285.143397] nix-grpc-daemon[1719]: time=2026-09-23T05:16:57.649Z level=INFO msg="Uploading 1 narinfos" lb # [ 285.114168] niks3-server[1111]: 2026/09/23 05:16:57 INFO Signed narinfos id=81 count=1 lb # [ 285.121222] niks3-server[1111]: 2026/09/23 05:16:57 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv lb # [ 285.128615] niks3-server[1111]: 2026/09/23 05:16:57 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst lb # [ 285.134629] niks3-server[1111]: 2026/09/23 05:16:57 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls lb # [ 285.145120] niks3-server[1111]: 2026/09/23 05:16:57 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete lb # [ 285.151967] niks3-server[1111]: 2026/09/23 05:16:57 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo lb # [ 285.156728] niks3-server[1111]: 2026/09/23 05:16:57 INFO Completed upload id=81 worker1 # [ 285.188957] nix-grpc-daemon[1719]: time=2026-09-23T05:16:57.695Z level=INFO msg="Upload complete. (308ms)" worker1 # [ 285.207145] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=2 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=10 outputs=1 worker1 # [ 285.228344] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:57Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 285.240424] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 285.249228] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:57Z level=debug event=want client=2 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=0 worker1 # [ 285.263718] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 285.274583] nix-daemon[832]: reaped child process 1984, status = succeeded worker1 # [ 285.279742] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 285.282934] nix-grpc-daemon[1719]: time=2026-09-23T05:16:57.752Z level=INFO msg="All 2 paths already cached" worker1 # [ 285.288422] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 285.411682] nix-daemon[832]: accepted connection from pid 2011, user nix-grpc-daemon (trusted) worker1 # [ 285.428131] nix-daemon[832]: reaped child process 2014, status = succeeded lb # [ 285.403854] niks3-server[1111]: 2026/09/23 05:16:57 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 285.449510] nix-grpc-daemon[1719]: time=2026-09-23T05:16:57.955Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 285.451079] nix-grpc-daemon[1719]: time=2026-09-23T05:16:57.955Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)" worker1 # [ 285.481588] nix-grpc-daemon[1719]: time=2026-09-23T05:16:57.987Z level=INFO msg="Uploading 1 narinfos" lb # [ 285.439395] niks3-server[1111]: 2026/09/23 05:16:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign lb # [ 285.457021] niks3-server[1111]: 2026/09/23 05:16:57 INFO Signed narinfos id=82 count=1 lb # [ 285.464785] niks3-server[1111]: 2026/09/23 05:16:57 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls lb # [ 285.476248] niks3-server[1111]: 2026/09/23 05:16:57 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst worker1 # [ 285.515126] nix-grpc-daemon[1719]: time=2026-09-23T05:16:58.021Z level=INFO msg="Upload complete. (232ms)" lb # [ 285.485394] niks3-server[1111]: 2026/09/23 05:16:57 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete lb # [ 285.492809] niks3-server[1111]: 2026/09/23 05:16:57 INFO Completed upload id=82 worker1 # [ 285.523739] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 285.495808] niks3-server[1111]: 2026/09/23 05:16:58 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo worker1 # [ 285.540525] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 285.548729] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 285.553903] nix-daemon[832]: reaped child process 2009, status = succeeded worker1 # [ 285.567519] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 287.240655] nix-daemon[832]: reaped child process 2018, status = succeeded worker1 # [ 287.348671] nix-daemon[832]: accepted connection from pid 2034, user nix-grpc-daemon (trusted) worker1 # [ 287.366730] nix-daemon[832]: reaped child process 2037, status = succeeded lb # [ 287.342230] niks3-server[1111]: 2026/09/23 05:16:59 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 287.384418] nix-grpc-daemon[1719]: time=2026-09-23T05:16:59.890Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 287.386111] nix-grpc-daemon[1719]: time=2026-09-23T05:16:59.890Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)" lb # [ 287.378367] niks3-server[1111]: 2026/09/23 05:16:59 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker1 # [ 287.437612] nix-grpc-daemon[1719]: time=2026-09-23T05:16:59.943Z level=INFO msg="Uploading 1 narinfos" lb # [ 287.400545] niks3-server[1111]: 2026/09/23 05:16:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign lb # [ 287.411340] niks3-server[1111]: 2026/09/23 05:16:59 INFO Signed narinfos id=83 count=1 lb # [ 287.416659] niks3-server[1111]: 2026/09/23 05:16:59 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst lb # [ 287.429545] niks3-server[1111]: 2026/09/23 05:16:59 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete worker1 # [ 287.460632] nix-grpc-daemon[1719]: time=2026-09-23T05:16:59.967Z level=INFO msg="Upload complete. (283ms)" lb # [ 287.433898] niks3-server[1111]: 2026/09/23 05:16:59 INFO Completed upload id=83 lb # [ 287.436530] niks3-server[1111]: 2026/09/23 05:16:59 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls lb # [ 287.440704] niks3-server[1111]: 2026/09/23 05:16:59 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo worker1 # [ 287.477102] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=11 outputs=1 worker1 # [ 287.493148] nix-grpc-daemon[1635]: ts=2026-09-23T05:16:59Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 287.498103] nix-daemon[832]: reaped child process 2016, status = succeeded worker1 # [ 287.517012] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 287.532442] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 287.547883] nix-grpc-daemon[1719]: time=2026-09-23T05:17:00.054Z level=INFO msg="All 2 paths already cached" worker1 # [ 287.558219] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 287.571848] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 287.720886] nix-daemon[832]: accepted connection from pid 2041, user nix-grpc-daemon (trusted) worker1 # [ 287.740713] nix-daemon[832]: reaped child process 2044, status = succeeded lb # [ 287.712907] niks3-server[1111]: 2026/09/23 05:17:00 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 287.754634] nix-grpc-daemon[1719]: time=2026-09-23T05:17:00.260Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 287.756433] nix-grpc-daemon[1719]: time=2026-09-23T05:17:00.260Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)" lb # [ 287.743437] niks3-server[1111]: 2026/09/23 05:17:00 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls lb # [ 287.755498] niks3-server[1111]: 2026/09/23 05:17:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign worker1 # [ 287.793913] nix-grpc-daemon[1719]: time=2026-09-23T05:17:00.300Z level=INFO msg="Uploading 1 narinfos" lb # [ 287.768373] niks3-server[1111]: 2026/09/23 05:17:00 INFO Signed narinfos id=84 count=1 lb # [ 287.773797] niks3-server[1111]: 2026/09/23 05:17:00 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst lb # [ 287.783331] niks3-server[1111]: 2026/09/23 05:17:00 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete lb # [ 287.789671] niks3-server[1111]: 2026/09/23 05:17:00 INFO Completed upload id=84 worker1 # [ 287.821464] nix-grpc-daemon[1719]: time=2026-09-23T05:17:00.328Z level=INFO msg="Upload complete. (234ms)" lb # [ 287.792587] niks3-server[1111]: 2026/09/23 05:17:00 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo worker1 # [ 287.832246] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 287.847859] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 287.854090] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 287.859077] nix-daemon[832]: reaped child process 2039, status = succeeded worker1 # [ 287.870087] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 289.144177] nix-daemon[832]: reaped child process 2048, status = succeeded worker1 # [ 289.245039] nix-daemon[832]: accepted connection from pid 2064, user nix-grpc-daemon (trusted) worker1 # [ 289.265765] nix-daemon[832]: reaped child process 2067, status = succeeded lb # [ 289.232906] niks3-server[1111]: 2026/09/23 05:17:01 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 289.277729] nix-grpc-daemon[1719]: time=2026-09-23T05:17:01.783Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 289.279362] nix-grpc-daemon[1719]: time=2026-09-23T05:17:01.783Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)" lb # [ 289.271709] niks3-server[1111]: 2026/09/23 05:17:01 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls worker1 # [ 289.318427] nix-grpc-daemon[1719]: time=2026-09-23T05:17:01.824Z level=INFO msg="Uploading 1 narinfos" lb # [ 289.285919] niks3-server[1111]: 2026/09/23 05:17:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign lb # [ 289.297961] niks3-server[1111]: 2026/09/23 05:17:01 INFO Signed narinfos id=85 count=1 lb # [ 289.304321] niks3-server[1111]: 2026/09/23 05:17:01 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv lb # [ 289.315374] niks3-server[1111]: 2026/09/23 05:17:01 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst worker1 # [ 289.354962] nix-grpc-daemon[1719]: time=2026-09-23T05:17:01.861Z level=INFO msg="Upload complete. (273ms)" lb # [ 289.323566] niks3-server[1111]: 2026/09/23 05:17:01 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete lb # [ 289.327440] niks3-server[1111]: 2026/09/23 05:17:01 INFO Completed upload id=85 lb # [ 289.331893] niks3-server[1111]: 2026/09/23 05:17:01 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo worker1 # [ 289.369388] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=12 outputs=1 worker1 # [ 289.384400] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:01Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 289.390213] nix-daemon[832]: reaped child process 2046, status = succeeded worker1 # [ 289.408901] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 289.423623] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 289.438496] nix-grpc-daemon[1719]: time=2026-09-23T05:17:01.945Z level=INFO msg="All 2 paths already cached" worker1 # [ 289.448593] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 289.461572] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 289.606304] nix-daemon[832]: accepted connection from pid 2071, user nix-grpc-daemon (trusted) worker1 # [ 289.622305] nix-daemon[832]: reaped child process 2074, status = succeeded lb # [ 289.598157] niks3-server[1111]: 2026/09/23 05:17:02 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 289.638881] nix-grpc-daemon[1719]: time=2026-09-23T05:17:02.145Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 289.640535] nix-grpc-daemon[1719]: time=2026-09-23T05:17:02.145Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)" worker1 # [ 289.659925] nix-grpc-daemon[1719]: time=2026-09-23T05:17:02.166Z level=INFO msg="Uploading 1 narinfos" lb # [ 289.627608] niks3-server[1111]: 2026/09/23 05:17:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/86/sign lb # [ 289.641745] niks3-server[1111]: 2026/09/23 05:17:02 INFO Signed narinfos id=86 count=1 lb # [ 289.649401] niks3-server[1111]: 2026/09/23 05:17:02 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst worker1 # [ 289.696188] nix-grpc-daemon[1719]: time=2026-09-23T05:17:02.202Z level=INFO msg="Upload complete. (219ms)" lb # [ 289.664559] niks3-server[1111]: 2026/09/23 05:17:02 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls lb # [ 289.670445] niks3-server[1111]: 2026/09/23 05:17:02 INFO Received complete upload request method=POST path=/api/pending_closures/86/complete lb # [ 289.680288] niks3-server[1111]: 2026/09/23 05:17:02 INFO Completed upload id=86 worker1 # [ 289.704121] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 289.682935] niks3-server[1111]: 2026/09/23 05:17:02 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo worker1 # [ 289.719004] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 289.725420] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 289.730475] nix-daemon[832]: reaped child process 2069, status = succeeded worker1 # [ 289.741406] nix-daemon[832]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 290.567259] nix-daemon[832]: reaped child process 2078, status = succeeded worker1 # [ 290.663669] nix-daemon[832]: accepted connection from pid 2094, user nix-grpc-daemon (trusted) worker1 # [ 290.679631] nix-daemon[832]: reaped child process 2097, status = succeeded lb # [ 290.658007] niks3-server[1111]: 2026/09/23 05:17:03 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 290.700080] nix-grpc-daemon[1719]: time=2026-09-23T05:17:03.204Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 290.701666] nix-grpc-daemon[1719]: time=2026-09-23T05:17:03.204Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)" lb # [ 290.699282] niks3-server[1111]: 2026/09/23 05:17:03 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker1 # [ 290.745240] nix-grpc-daemon[1719]: time=2026-09-23T05:17:03.251Z level=INFO msg="Uploading 1 narinfos" lb # [ 290.715053] niks3-server[1111]: 2026/09/23 05:17:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/87/sign lb # [ 290.725803] niks3-server[1111]: 2026/09/23 05:17:03 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls lb # [ 290.730633] niks3-server[1111]: 2026/09/23 05:17:03 INFO Signed narinfos id=87 count=1 lb # [ 290.739147] niks3-server[1111]: 2026/09/23 05:17:03 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst worker1 # [ 290.776788] nix-grpc-daemon[1719]: time=2026-09-23T05:17:03.283Z level=INFO msg="Upload complete. (224ms)" lb # [ 290.746653] niks3-server[1111]: 2026/09/23 05:17:03 INFO Received complete upload request method=POST path=/api/pending_closures/87/complete lb # [ 290.753629] niks3-server[1111]: 2026/09/23 05:17:03 INFO Completed upload id=87 lb # [ 290.757454] niks3-server[1111]: 2026/09/23 05:17:03 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo worker1 # [ 290.789018] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55690 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=13 outputs=1 worker1 # [ 290.805246] nix-daemon[832]: reaped child process 2076, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 17.49 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]' client # [ 291.020997] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]". client # Running as unit: gpu.service; invocation ID: 556ad90cda7d4022bd3d9bc537d5b000 client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.10 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker1 # [ 291.207763] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55690 client # [ 291.234240] nix[2066]: don't know how to build these paths: client # [ 291.235202] nix[2066]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv worker2 # [ 291.287755] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47332 client # [ 291.265897] nix[2066]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip}, waiting up to 30s for a worker worker1 # [ 291.245103] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:03Z level=debug event=want client=3 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) client: waiting for success: systemctl show -p Result gpu | grep -q exit-code lb # [ 299.203484] envoy[895]: conn peer=[2001:db8:1::1]:45676 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27226 tx=12434 ms=17223 lb # [ 315.538186] postgres[798]: [798] LOG: checkpoint starting: time client # [ 321.267438] nix[2066]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip} client # [ 321.285392] nix[2066]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed client # [ 321.317761] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE client # [ 321.326079] systemd[1]: gpu.service: Failed with result 'exit-code'. client # [ 321.330000] systemd[1]: gpu.service: Consumed 202ms CPU time over 30.303s wall clock time, 37.4M memory peak, 5.6K incoming IP traffic, 5.3K outgoing IP traffic. client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.07 seconds) client: must succeed: journalctl -u gpu -o cat client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 48.29 seconds) subtest: low disk drains a worker and builds go to the other worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.14 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free' worker1 # [ 323.067643] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:35Z level=info event=unhealthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.10 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2 client # don't know how to build these paths: client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker1 # [ 324.136994] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:36Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 324.144620] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55690 worker1 # [ 324.168124] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:36Z level=debug event=want client=4 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=0 worker1 # [ 324.174609] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:36Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 324.291503] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 324.308797] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47332 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'... worker2 # [ 324.324711] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 324.342717] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 324.518912] nix-daemon[855]: accepted connection from pid 2049, user nix-grpc-daemon (trusted) worker2 # [ 324.542534] nix-daemon[855]: reaped child process 2052, status = succeeded lb # [ 324.442181] niks3-server[1111]: 2026/09/23 05:17:36 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 324.552442] nix-grpc-daemon[1805]: time=2026-09-23T05:17:37.005Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 324.553946] nix-grpc-daemon[1805]: time=2026-09-23T05:17:37.005Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)" lb # [ 324.483111] niks3-server[1111]: 2026/09/23 05:17:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/88/sign worker2 # [ 324.592591] nix-grpc-daemon[1805]: time=2026-09-23T05:17:37.045Z level=INFO msg="Uploading 1 narinfos" lb # [ 324.494896] niks3-server[1111]: 2026/09/23 05:17:37 INFO Signed narinfos id=88 count=1 lb # [ 324.502865] niks3-server[1111]: 2026/09/23 05:17:37 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls lb # [ 324.513661] niks3-server[1111]: 2026/09/23 05:17:37 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst lb # [ 324.526895] niks3-server[1111]: 2026/09/23 05:17:37 INFO Received complete upload request method=POST path=/api/pending_closures/88/complete lb # [ 324.532296] niks3-server[1111]: 2026/09/23 05:17:37 INFO Completed upload id=88 worker2 # [ 324.632066] nix-grpc-daemon[1805]: time=2026-09-23T05:17:37.085Z level=INFO msg="Upload complete. (267ms)" lb # [ 324.539949] niks3-server[1111]: 2026/09/23 05:17:37 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo worker2 # [ 324.642335] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 324.659991] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 324.669624] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 324.672483] nix-daemon[855]: reaped child process 2047, status = succeeded client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051... worker2 # [ 324.685633] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) client # farm-a> worker2: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-drain worker2 # [ 325.914823] nix-daemon[855]: reaped child process 2056, status = succeeded worker2 # [ 326.024393] nix-daemon[855]: accepted connection from pid 2072, user nix-grpc-daemon (trusted) worker2 # [ 326.045248] nix-daemon[855]: reaped child process 2075, status = succeeded lb # [ 325.950016] niks3-server[1111]: 2026/09/23 05:17:38 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 326.062931] nix-grpc-daemon[1805]: time=2026-09-23T05:17:38.514Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 326.064527] nix-grpc-daemon[1805]: time=2026-09-23T05:17:38.514Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)" lb # [ 325.992145] niks3-server[1111]: 2026/09/23 05:17:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/89/sign worker2 # [ 326.100083] nix-grpc-daemon[1805]: time=2026-09-23T05:17:38.553Z level=INFO msg="Uploading 1 narinfos" lb # [ 326.005707] niks3-server[1111]: 2026/09/23 05:17:38 INFO Signed narinfos id=89 count=1 lb # [ 326.012420] niks3-server[1111]: 2026/09/23 05:17:38 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst lb # [ 326.024225] niks3-server[1111]: 2026/09/23 05:17:38 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker2 # [ 326.132979] nix-grpc-daemon[1805]: time=2026-09-23T05:17:38.586Z level=INFO msg="Upload complete. (238ms)" lb # [ 326.033065] niks3-server[1111]: 2026/09/23 05:17:38 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls lb # [ 326.038042] niks3-server[1111]: 2026/09/23 05:17:38 INFO Received complete upload request method=POST path=/api/pending_closures/89/complete lb # [ 326.043127] niks3-server[1111]: 2026/09/23 05:17:38 INFO Completed upload id=89 worker2 # [ 326.144955] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=14 outputs=1 lb # [ 326.047688] niks3-server[1111]: 2026/09/23 05:17:38 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo worker1 # [ 326.081939] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:38Z level=debug event=want client=4 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=0 worker2 # [ 326.161015] nix-daemon[855]: reaped child process 2054, status = succeeded worker1 # [ 326.088005] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:38Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 326.198286] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 326.213360] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 326.228100] nix-grpc-daemon[1805]: time=2026-09-23T05:17:38.681Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'... worker2 # [ 326.239441] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 326.254800] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 326.405641] nix-daemon[855]: accepted connection from pid 2079, user nix-grpc-daemon (trusted) worker2 # [ 326.422469] nix-daemon[855]: reaped child process 2082, status = succeeded lb # [ 326.328039] niks3-server[1111]: 2026/09/23 05:17:38 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 326.438631] nix-grpc-daemon[1805]: time=2026-09-23T05:17:38.891Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 326.440349] nix-grpc-daemon[1805]: time=2026-09-23T05:17:38.891Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)" lb # [ 326.355686] niks3-server[1111]: 2026/09/23 05:17:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/90/sign worker2 # [ 326.463165] nix-grpc-daemon[1805]: time=2026-09-23T05:17:38.916Z level=INFO msg="Uploading 1 narinfos" lb # [ 326.367931] niks3-server[1111]: 2026/09/23 05:17:38 INFO Signed narinfos id=90 count=1 lb # [ 326.374448] niks3-server[1111]: 2026/09/23 05:17:38 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls lb # [ 326.386220] niks3-server[1111]: 2026/09/23 05:17:38 INFO Received complete upload request method=POST path=/api/pending_closures/90/complete worker2 # [ 326.492432] nix-grpc-daemon[1805]: time=2026-09-23T05:17:38.945Z level=INFO msg="Upload complete. (220ms)" lb # [ 326.391515] niks3-server[1111]: 2026/09/23 05:17:38 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst lb # [ 326.396594] niks3-server[1111]: 2026/09/23 05:17:38 INFO Completed upload id=90 lb # [ 326.399110] niks3-server[1111]: 2026/09/23 05:17:38 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo worker2 # [ 326.500110] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=0 paths=1 nar_bytes_in=632 worker2 # [ 326.513589] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 326.519362] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 326.523972] nix-daemon[855]: reaped child process 2077, status = succeeded client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051... worker2 # [ 326.534857] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) client # farm-b> worker2: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-drain worker2 # [ 327.401843] nix-daemon[855]: reaped child process 2086, status = succeeded worker2 # [ 327.514914] nix-daemon[855]: accepted connection from pid 2102, user nix-grpc-daemon (trusted) worker2 # [ 327.532198] nix-daemon[855]: reaped child process 2105, status = succeeded lb # [ 327.438263] niks3-server[1111]: 2026/09/23 05:17:39 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 327.552414] nix-grpc-daemon[1805]: time=2026-09-23T05:17:40.002Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 327.555590] nix-grpc-daemon[1805]: time=2026-09-23T05:17:40.002Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)" lb # [ 327.476292] niks3-server[1111]: 2026/09/23 05:17:39 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv lb # [ 327.491820] niks3-server[1111]: 2026/09/23 05:17:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/91/sign worker2 # [ 327.601217] nix-grpc-daemon[1805]: time=2026-09-23T05:17:40.054Z level=INFO msg="Uploading 1 narinfos" lb # [ 327.505602] niks3-server[1111]: 2026/09/23 05:17:40 INFO Signed narinfos id=91 count=1 lb # [ 327.512371] niks3-server[1111]: 2026/09/23 05:17:40 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls lb # [ 327.523036] niks3-server[1111]: 2026/09/23 05:17:40 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst worker2 # [ 327.631351] nix-grpc-daemon[1805]: time=2026-09-23T05:17:40.084Z level=INFO msg="Upload complete. (282ms)" lb # [ 327.532548] niks3-server[1111]: 2026/09/23 05:17:40 INFO Received complete upload request method=POST path=/api/pending_closures/91/complete lb # [ 327.539011] niks3-server[1111]: 2026/09/23 05:17:40 INFO Completed upload id=91 lb # [ 327.541703] niks3-server[1111]: 2026/09/23 05:17:40 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo worker2 # [ 327.645160] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=15 outputs=1 worker1 # [ 327.582082] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:40Z level=debug event=want client=4 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=0 worker1 # [ 327.587864] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:40Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 327.664158] nix-daemon[855]: reaped child process 2084, status = succeeded worker2 # [ 327.698045] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 327.712915] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 327.726590] nix-grpc-daemon[1805]: time=2026-09-23T05:17:40.179Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 327.737474] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'... worker2 # [ 327.752358] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 327.907372] nix-daemon[855]: accepted connection from pid 2109, user nix-grpc-daemon (trusted) worker2 # [ 327.924257] nix-daemon[855]: reaped child process 2112, status = succeeded lb # [ 327.828488] niks3-server[1111]: 2026/09/23 05:17:40 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 327.939545] nix-grpc-daemon[1805]: time=2026-09-23T05:17:40.392Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 327.941323] nix-grpc-daemon[1805]: time=2026-09-23T05:17:40.392Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)" lb # [ 327.857966] niks3-server[1111]: 2026/09/23 05:17:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/92/sign worker2 # [ 327.966258] nix-grpc-daemon[1805]: time=2026-09-23T05:17:40.419Z level=INFO msg="Uploading 1 narinfos" lb # [ 327.870505] niks3-server[1111]: 2026/09/23 05:17:40 INFO Signed narinfos id=92 count=1 lb # [ 327.879986] niks3-server[1111]: 2026/09/23 05:17:40 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls lb # [ 327.895755] niks3-server[1111]: 2026/09/23 05:17:40 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst worker2 # [ 328.008815] nix-grpc-daemon[1805]: time=2026-09-23T05:17:40.461Z level=INFO msg="Upload complete. (238ms)" lb # [ 327.906423] niks3-server[1111]: 2026/09/23 05:17:40 INFO Received complete upload request method=POST path=/api/pending_closures/92/complete lb # [ 327.911295] niks3-server[1111]: 2026/09/23 05:17:40 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo lb # [ 327.917894] niks3-server[1111]: 2026/09/23 05:17:40 INFO Completed upload id=92 worker2 # [ 328.017638] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=0 paths=1 nar_bytes_in=760 worker2 # [ 328.033584] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 worker2 # [ 328.040484] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) worker2 # [ 328.046033] nix-daemon[855]: reaped child process 2107, status = succeeded client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051... worker2 # [ 328.058193] nix-daemon[855]: accepted connection from pid 1723, user nix-grpc-daemon (trusted) client # farm-top> worker2: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-drain worker2 # [ 329.164636] nix-daemon[855]: reaped child process 2116, status = succeeded worker2 # [ 329.236901] nix-daemon[855]: accepted connection from pid 2132, user nix-grpc-daemon (trusted) worker2 # [ 329.257214] nix-daemon[855]: reaped child process 2135, status = succeeded lb # [ 329.161242] niks3-server[1111]: 2026/09/23 05:17:41 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 329.274034] nix-grpc-daemon[1805]: time=2026-09-23T05:17:41.726Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 329.275739] nix-grpc-daemon[1805]: time=2026-09-23T05:17:41.726Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)" lb # [ 329.204360] niks3-server[1111]: 2026/09/23 05:17:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/93/sign worker2 # [ 329.312660] nix-grpc-daemon[1805]: time=2026-09-23T05:17:41.765Z level=INFO msg="Uploading 1 narinfos" lb # [ 329.218248] niks3-server[1111]: 2026/09/23 05:17:41 INFO Signed narinfos id=93 count=1 lb # [ 329.228809] envoy[895]: conn peer=[2001:db8:1::1]:46908 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=931 tx=613 ms=30067 lb # [ 329.236295] niks3-server[1111]: 2026/09/23 05:17:41 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst lb # [ 329.245663] niks3-server[1111]: 2026/09/23 05:17:41 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker2 # [ 329.352133] nix-grpc-daemon[1805]: time=2026-09-23T05:17:41.805Z level=INFO msg="Upload complete. (280ms)" lb # [ 329.251412] niks3-server[1111]: 2026/09/23 05:17:41 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls lb # [ 329.255526] niks3-server[1111]: 2026/09/23 05:17:41 INFO Received complete upload request method=POST path=/api/pending_closures/93/complete lb # [ 329.262632] niks3-server[1111]: 2026/09/23 05:17:41 INFO Completed upload id=93 worker2 # [ 329.363327] nix-grpc-daemon[1723]: ts=2026-09-23T05:17:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:47332 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=16 outputs=1 lb # [ 329.265872] niks3-server[1111]: 2026/09/23 05:17:41 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo worker2 # [ 329.383029] nix-daemon[855]: reaped child process 2114, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 5.43 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.18 seconds) worker1: must succeed: rm /nix/.rw-store/fill worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.16 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free' worker1 # [ 330.069917] nix-grpc-daemon[1635]: ts=2026-09-23T05:17:42Z level=info event=healthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.10 seconds) (finished: subtest: low disk drains a worker and builds go to the other, in 9.21 seconds) (finished: run the VM test script, in 331.87 seconds) test script finished in 332.12s cleanup kill QemuMachine (pid 49) client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) client # [2026-09-23T05:17:43Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-23T05:17:43Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-23T05:17:43Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 50) lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) lb # [2026-09-23T05:17:44Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-23T05:17:44Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-23T05:17:44Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 51) worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker1 # [2026-09-23T05:17:44Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-23T05:17:44Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-23T05:17:44Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 52) worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker2 # [2026-09-23T05:17:45Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-23T05:17:45Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-23T05:17:45Z INFO virtiofsd] Client disconnected, shutting down (finished: cleanup, in 1.61 seconds) additionally exposed symbols: client, lb, worker1, worker2, vlan1, start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh