tribuchet: building on eliza Machine state will be reset. To keep it, pass --keep-machine-state start all VLans (finished: start all VLans, in 0.00 seconds) Test will time out and terminate in 900.0 seconds run the VM test script start all VMs client: starting vm lb: starting vm worker1: starting vm worker2: starting vm client # Disk image does not exist, creating the virtualisation disk image... lb # Disk image does not exist, creating the virtualisation disk image... lb: QEMU running (pid 50) client: QEMU running (pid 49) client # Formatting '/build/vm-state-client/tmp.IMSBbWYzp9', fmt=raw size=1073741824 worker1: QEMU running (pid 51) worker2 # Disk image does not exist, creating the virtualisation disk image... worker1 # Disk image does not exist, creating the virtualisation disk image... lb # Formatting '/build/vm-state-lb/tmp.Rsg3X9rR5k', fmt=raw size=1073741824 worker2 # Formatting '/build/vm-state-worker2/tmp.vV31HA7IFp', fmt=raw size=1073741824 worker1 # Formatting '/build/vm-state-worker1/tmp.Mx5rt263li', fmt=raw size=1073741824 lb # mke2fs 1.47.4 (6-Mar-2025) worker2 # mke2fs 1.47.4 (6-Mar-2025) worker1 # mke2fs 1.47.4 (6-Mar-2025) lb # Discarding device blocks: 0/262144 done worker2 # Discarding device blocks: 0/262144 done worker1 # Discarding device blocks: 0/262144 done lb # Creating filesystem with 262144 4k blocks and 65536 inodes lb # Filesystem UUID: 80d3eab8-8922-4532-88a9-c338635f91f5 worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes client # mke2fs 1.47.4 (6-Mar-2025) worker2: QEMU running (pid 52) worker2 # Filesystem UUID: 32fcf0a6-95eb-4478-af00-18bcbf63d3d3 client # Discarding device blocks: 0/262144 done worker2 # Superblock backups stored on blocks: client # Creating filesystem with 262144 4k blocks and 65536 inodes worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes worker2 # 32768, 98304, 163840, 229376 client # Filesystem UUID: 80cc710e-c6b4-4276-acd8-4081619fb465 worker1 # Filesystem UUID: d9b10c6c-1981-4594-a993-de61abb4c086 worker2 # client # Superblock backups stored on blocks: worker1 # Superblock backups stored on blocks: worker2 # Allocating group tables: 0/8 done client # 32768, 98304, 163840, 229376 worker1 # 32768, 98304, 163840, 229376 worker2 # Writing inode tables: 0/8 done client # worker1 # worker2 # Creating journal (8192 blocks): done client # Allocating group tables: 0/8 done worker1 # Allocating group tables: 0/8 done worker2 # Writing superblocks and filesystem accounting information: 0/8 done client # Writing inode tables: 0/8 done worker1 # Writing inode tables: 0/8 done worker2 # client # Creating journal (8192 blocks): done worker1 # Creating journal (8192 blocks): done worker2 # Virtualisation disk image created. client # Writing superblocks and filesystem accounting information: 0/8 done worker1 # Writing superblocks and filesystem accounting information: 0/8 done worker2 # Starting virtiofs daemons... client # worker1 # worker2 # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # Virtualisation disk image created. worker1 # Virtualisation disk image created. worker2 # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # Starting virtiofs daemons... worker1 # Starting virtiofs daemons... worker2 # [2026-09-25T14:59:20Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-25T14:59:20Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-25T14:59:20Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-25T14:59:20Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-25T14:59:20Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-25T14:59:20Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-25T14:59:20Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-25T14:59:20Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-25T14:59:20Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-25T14:59:20Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-25T14:59:20Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-25T14:59:20Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-25T14:59:20Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-25T14:59:20Z INFO virtiofsd] Client connected, servicing requests (finished: start all VMs, in 0.54 seconds) lb # Superblock backups stored on blocks: client # [2026-09-25T14:59:20Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-25T14:59:20Z INFO virtiofsd] Client connected, servicing requests client: waiting for unit network-addresses-eth1.service lb # 32768, 98304, 163840, 229376 client # [2026-09-25T14:59:20Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-25T14:59:20Z INFO virtiofsd] Client connected, servicing requests client: waiting for the VM to finish booting lb # lb # Allocating group tables: 0/8 done lb # Writing inode tables: 0/8 done lb # Creating journal (8192 blocks): done lb # Writing superblocks and filesystem accounting information: 0/8 done lb # lb # Virtualisation disk image created. lb # Starting virtiofs daemons... lb # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-25T14:59:20Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-25T14:59:20Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-25T14:59:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-25T14:59:20Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-25T14:59:20Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-25T14:59:20Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-25T14:59:20Z INFO virtiofsd] Client connected, servicing requests worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker1 # [ 0.000000] KASLR enabled worker1 # [ 0.000000] random: crng init done worker1 # [ 0.000000] Machine model: linux,dummy-virt worker1 # [ 0.000000] efi: UEFI not found. worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker1 # [ 0.000000] Zone ranges: worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] DMA32 empty worker1 # [ 0.000000] Normal empty worker1 # [ 0.000000] Device empty worker1 # [ 0.000000] Movable zone start for each node worker1 # [ 0.000000] Early memory node ranges worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker1 # [ 0.000000] psci: probing for conduit method from DT. worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker1 # [ 0.000000] psci: Trusted OS migration not required worker1 # [ 0.000000] psci: SMC Calling Convention v1.1 worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker1 # [ 0.000000] Detected PIPT I-cache on CPU0 worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface worker1 # [ 0.000000] CPU features: detected: Spectre-v4 worker1 # [ 0.000000] CPU features: detected: Spectre-BHB worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker1 # [ 0.000000] alternatives: applying boot alternatives client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 client # [ 0.000000] KASLR enabled lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 client # [ 0.000000] random: crng init done lb # [ 0.000000] KASLR enabled client # [ 0.000000] Machine model: linux,dummy-virt lb # [ 0.000000] random: crng init done worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/r1caxc5yfxdfis0l5rpgjh307ha4z8wh-nixos-system-worker1-test/init regInfo=/nix/store/b307fgar6jj2m9w47c3b4gq7jgva1agm-closure-info/registration console=ttyAMA0,115200n8 console=tty0 client # [ 0.000000] efi: UEFI not found. lb # [ 0.000000] Machine model: linux,dummy-virt worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] lb # [ 0.000000] efi: UEFI not found. client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/b307fgar6jj2m9w47c3b4gq7jgva1agm-closure-info/registration", will be passed to user space. client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker2 # [ 0.000000] KASLR enabled lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes client # [ 0.000000] Zone ranges: worker2 # [ 0.000000] random: crng init done lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker2 # [ 0.000000] Machine model: linux,dummy-virt worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) lb # [ 0.000000] Zone ranges: client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] efi: UEFI not found. client # [ 0.000000] DMA32 empty client # [ 0.000000] Normal empty lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) client # [ 0.000000] Device empty lb # [ 0.000000] DMA32 empty worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT lb # [ 0.000000] Normal empty client # [ 0.000000] Movable zone start for each node worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB lb # [ 0.000000] Device empty worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] Early memory node ranges worker1 # [ 0.000000] software IO TLB: area num 1. lb # [ 0.000000] Movable zone start for each node worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] lb # [ 0.000000] Early memory node ranges client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker2 # [ 0.000000] Zone ranges: lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker2 # [ 0.000000] DMA32 empty client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker1 # [ 0.000000] Fallback order for Node 0: 0 lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] Normal empty worker2 # [ 0.000000] Device empty client # [ 0.000000] psci: probing for conduit method from DT. lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker2 # [ 0.000000] Movable zone start for each node client # [ 0.000000] psci: PSCIv1.3 detected in firmware. lb # [ 0.000000] psci: probing for conduit method from DT. worker1 # [ 0.000000] Policy zone: DMA worker2 # [ 0.000000] Early memory node ranges client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs lb # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] psci: Trusted OS migration not required lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs client # [ 0.000000] psci: SMC Calling Convention v1.1 worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 lb # [ 0.000000] psci: Trusted OS migration not required worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] allocated 3145728 bytes of page_ext lb # [ 0.000000] psci: SMC Calling Convention v1.1 worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages worker2 # [ 0.000000] psci: probing for conduit method from DT. lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware. client # [ 0.000000] Detected PIPT I-cache on CPU0 worker1 # [ 0.000000] rcu: Hierarchical RCU implementation. lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker1 # [ 0.000000] rcu: RCU event tracing is enabled. lb # [ 0.000000] Detected PIPT I-cache on CPU0 worker2 # [ 0.000000] psci: Trusted OS migration not required client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker2 # [ 0.000000] psci: SMC Calling Convention v1.1 worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) client # [ 0.000000] CPU features: detected: GICv3 CPU interface worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled. client # [ 0.000000] CPU features: detected: Spectre-v4 lb # [ 0.000000] CPU features: detected: GICv3 CPU interface worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.000000] Rude variant of Tasks RCU enabled. client # [ 0.000000] CPU features: detected: Spectre-BHB lb # [ 0.000000] CPU features: detected: Spectre-v4 worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled. worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 lb # [ 0.000000] CPU features: detected: Spectre-BHB client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker2 # [ 0.000000] Detected PIPT I-cache on CPU0 worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 client # [ 0.000000] alternatives: applying boot alternatives worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface lb # [ 0.000000] alternatives: applying boot alternatives worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] CPU features: detected: Spectre-v4 worker2 # [ 0.000000] CPU features: detected: Spectre-BHB worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/jgg1bkcnnwpis3sdm7gl6kzqxlwc5ivn-nixos-system-client-test/init regInfo=/nix/store/mlssl3r6fkrygq3m0y463sc9vyikkhk0-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker2 # [ 0.000000] alternatives: applying boot alternatives worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker1 # [ 0.000000] GICv3: 256 SPIs implemented lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/5hmvppg1fs8xkiccvl0m50n4dxjz92fr-nixos-system-lb-test/init regInfo=/nix/store/4s18yqhkjka9yjskv51fw7r9nw16dgfd-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/mlssl3r6fkrygq3m0y463sc9vyikkhk0-closure-info/registration", will be passed to user space. worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/4s18yqhkjka9yjskv51fw7r9nw16dgfd-closure-info/registration", will be passed to user space. client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/azx0l63xg7ndaj4b8gzinjsgk55i1kf6-nixos-system-worker2-test/init regInfo=/nix/store/wnigwk8c8nbnp7lb6ngki07q6kjn1xls-closure-info/registration console=ttyAMA0,115200n8 console=tty0 lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/wnigwk8c8nbnp7lb6ngki07q6kjn1xls-closure-info/registration", will be passed to user space. client # [ 0.000000] software IO TLB: area num 1. lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB lb # [ 0.000000] software IO TLB: area num 1. worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) client # [ 0.000000] Fallback order for Node 0: 0 lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) lb # [ 0.000000] Fallback order for Node 0: 0 client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB client # [ 0.000000] Policy zone: DMA lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker2 # [ 0.000000] software IO TLB: area num 1. lb # [ 0.000000] Policy zone: DMA client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 client # [ 0.000000] allocated 3145728 bytes of page_ext worker2 # [ 0.000000] Fallback order for Node 0: 0 lb # [ 0.000000] allocated 3145728 bytes of page_ext client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 client # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker2 # [ 0.000000] Policy zone: DMA lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups client # [ 0.000000] rcu: Hierarchical RCU implementation. worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker1 # [ 0.000037] arm-pv: using stolen time PV lb # [ 0.000000] rcu: Hierarchical RCU implementation. worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off client # [ 0.000000] rcu: RCU event tracing is enabled. lb # [ 0.000000] rcu: RCU event tracing is enabled. worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker1 # [ 0.000708] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker2 # [ 0.000000] allocated 3145728 bytes of page_ext lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. client # [ 0.000000] Trampoline variant of Tasks RCU enabled. worker1 # [ 0.000934] Console: colour dummy device 80x25 worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages lb # [ 0.000000] Trampoline variant of Tasks RCU enabled. client # [ 0.000000] Rude variant of Tasks RCU enabled. worker1 # [ 0.000942] printk: legacy console [tty0] enabled lb # [ 0.000000] Rude variant of Tasks RCU enabled. worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups client # [ 0.000000] Tracing variant of Tasks RCU enabled. lb # [ 0.000000] Tracing variant of Tasks RCU enabled. worker2 # [ 0.000000] rcu: Hierarchical RCU implementation. client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker1 # [ 0.001133] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker2 # [ 0.000000] rcu: RCU event tracing is enabled. lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker1 # [ 0.001140] pid_max: default: 32768 minimum: 301 client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 worker1 # [ 0.001228] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled. worker1 # [ 0.001421] landlock: Up and running. client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] Rude variant of Tasks RCU enabled. worker1 # [ 0.001425] Yama: becoming mindful. lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.002088] LSM support for eBPF active client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker1 # [ 0.002248] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.002284] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 client # [ 0.000000] GICv3: 256 SPIs implemented lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.000000] GICv3: 0 Extended SPIs implemented lb # [ 0.000000] GICv3: 256 SPIs implemented worker1 # [ 0.004417] rcu: Hierarchical SRCU implementation. client # [ 0.000000] Root IRQ handler: gic_handle_irq lb # [ 0.000000] GICv3: 0 Extended SPIs implemented worker1 # [ 0.004423] rcu: Max phase no-delay instances is 1000. worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] Root IRQ handler: gic_handle_irq client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI worker1 # [ 0.005745] fsl-mc MSI: its@8080000 domain created lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker1 # [ 0.005832] EFI services will not be available. worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker1 # [ 0.005912] smp: Bringing up secondary CPUs ... client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker1 # [ 0.005922] smp: Brought up 1 node, 1 CPU client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker2 # [ 0.000000] GICv3: 256 SPIs implemented worker1 # [ 0.005925] SMP: Total of 1 processors activated. lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented worker1 # [ 0.005930] CPU: All CPU(s) started at EL1 worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI worker1 # [ 0.005941] CPU features: detected: Branch Target Identification worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker1 # [ 0.005946] CPU features: detected: ARMv8.4 Translation Table Level lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker1 # [ 0.005949] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker1 # [ 0.005953] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. worker1 # [ 0.005957] CPU features: detected: Common not Private translations worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker1 # [ 0.005960] CPU features: detected: CRC32 instructions client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker1 # [ 0.005963] CPU features: detected: Data cache clean to Point of Deep Persistence lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker1 # [ 0.005966] CPU features: detected: Data cache clean to Point of Persistence client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker1 # [ 0.005969] CPU features: detected: Data independent timing control (DIT) client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker1 # [ 0.005973] CPU features: detected: E0PD lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns client # [ 0.000032] arm-pv: using stolen time PV worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker1 # [ 0.005975] CPU features: detected: Enhanced Counter Virtualization lb # [ 0.000032] arm-pv: using stolen time PV worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker1 # [ 0.005978] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) client # [ 0.000381] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) lb # [ 0.000417] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) client # [ 0.000571] Console: colour dummy device 80x25 worker1 # [ 0.005981] CPU features: detected: Enhanced Virtualization Traps worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns lb # [ 0.000617] Console: colour dummy device 80x25 client # [ 0.000579] printk: legacy console [tty0] enabled worker1 # [ 0.005984] CPU features: detected: Fine Grained Traps lb # [ 0.000627] printk: legacy console [tty0] enabled worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker1 # [ 0.005988] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker2 # [ 0.000031] arm-pv: using stolen time PV client # [ 0.000778] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) lb # [ 0.000817] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker1 # [ 0.005992] CPU features: detected: RCpc load-acquire (LDAPR) client # [ 0.000784] pid_max: default: 32768 minimum: 301 lb # [ 0.000824] pid_max: default: 32768 minimum: 301 worker2 # [ 0.000404] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker1 # [ 0.005995] CPU features: detected: LSE atomic instructions client # [ 0.000871] LSM: initializing lsm=capability,landlock,yama,bpf,ima lb # [ 0.000929] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker2 # [ 0.000573] Console: colour dummy device 80x25 client # [ 0.001055] landlock: Up and running. worker1 # [ 0.005998] CPU features: detected: Privileged Access Never lb # [ 0.001095] landlock: Up and running. client # [ 0.001058] Yama: becoming mindful. worker2 # [ 0.000581] printk: legacy console [tty0] enabled worker1 # [ 0.006001] CPU features: detected: PMUv3 lb # [ 0.001099] Yama: becoming mindful. client # [ 0.001532] LSM support for eBPF active worker1 # [ 0.006004] CPU features: detected: RAS Extension Support lb # [ 0.001532] LSM support for eBPF active worker2 # [ 0.000784] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) client # [ 0.001679] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker1 # [ 0.006007] CPU features: detected: RASv1p1 Extension Support lb # [ 0.001699] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.000792] pid_max: default: 32768 minimum: 301 client # [ 0.001714] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker1 # [ 0.006009] CPU features: detected: Random Number Generator worker2 # [ 0.000869] LSM: initializing lsm=capability,landlock,yama,bpf,ima lb # [ 0.001733] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.003780] rcu: Hierarchical SRCU implementation. worker1 # [ 0.006012] CPU features: detected: Speculation barrier (SB) worker2 # [ 0.001004] landlock: Up and running. lb # [ 0.003709] rcu: Hierarchical SRCU implementation. client # [ 0.003786] rcu: Max phase no-delay instances is 1000. worker2 # [ 0.001006] Yama: becoming mindful. worker1 # [ 0.006014] CPU features: detected: Stage-2 Force Write-Back lb # [ 0.003715] rcu: Max phase no-delay instances is 1000. worker2 # [ 0.001468] LSM support for eBPF active client # [ 0.005138] fsl-mc MSI: its@8080000 domain created worker1 # [ 0.006017] CPU features: detected: TLB range maintenance instructions lb # [ 0.005023] fsl-mc MSI: its@8080000 domain created client # [ 0.005229] EFI services will not be available. worker2 # [ 0.001608] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) lb # [ 0.005124] EFI services will not be available. client # [ 0.005351] smp: Bringing up secondary CPUs ... worker1 # [ 0.006021] CPU features: detected: Speculative Store Bypassing Safe (SSBS) lb # [ 0.005220] smp: Bringing up secondary CPUs ... client # [ 0.005360] smp: Brought up 1 node, 1 CPU worker2 # [ 0.001644] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker1 # [ 0.006063] alternatives: applying system-wide alternatives lb # [ 0.005230] smp: Brought up 1 node, 1 CPU client # [ 0.005363] SMP: Total of 1 processors activated. worker2 # [ 0.003691] rcu: Hierarchical SRCU implementation. lb # [ 0.005233] SMP: Total of 1 processors activated. client # [ 0.005369] CPU: All CPU(s) started at EL1 worker1 # [ 0.009115] CPU features: detected: BBM Level 2 without TLB conflict abort worker2 # [ 0.003697] rcu: Max phase no-delay instances is 1000. lb # [ 0.005238] CPU: All CPU(s) started at EL1 client # [ 0.005380] CPU features: detected: Branch Target Identification worker2 # [ 0.005019] fsl-mc MSI: its@8080000 domain created lb # [ 0.005250] CPU features: detected: Branch Target Identification worker2 # [ 0.005111] EFI services will not be available. client # [ 0.005385] CPU features: detected: ARMv8.4 Translation Table Level worker1 # [ 0.009359] Memory: 1405588K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved) lb # [ 0.005254] CPU features: detected: ARMv8.4 Translation Table Level worker2 # [ 0.005232] smp: Bringing up secondary CPUs ... worker1 # [ 0.009813] devtmpfs: initialized worker2 # [ 0.005240] smp: Brought up 1 node, 1 CPU client # [ 0.005387] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker1 # [ 0.011980] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker2 # [ 0.005244] SMP: Total of 1 processors activated. lb # [ 0.005257] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker2 # [ 0.005249] CPU: All CPU(s) started at EL1 client # [ 0.005391] CPU features: detected: Data cache clean to the PoU not required for I/D coherence lb # [ 0.005261] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker1 # [ 0.012005] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker2 # [ 0.005261] CPU features: detected: Branch Target Identification client # [ 0.005395] CPU features: detected: Common not Private translations lb # [ 0.005264] CPU features: detected: Common not Private translations worker1 # [ 0.012209] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL client # [ 0.005398] CPU features: detected: CRC32 instructions worker2 # [ 0.005264] CPU features: detected: ARMv8.4 Translation Table Level worker1 # [ 0.012213] 0 pages in range for non-PLT usage lb # [ 0.005268] CPU features: detected: CRC32 instructions client # [ 0.005401] CPU features: detected: Data cache clean to Point of Deep Persistence worker1 # [ 0.012214] 508272 pages in range for PLT usage worker2 # [ 0.005267] CPU features: detected: Instruction cache invalidation not required for I/D coherence lb # [ 0.005270] CPU features: detected: Data cache clean to Point of Deep Persistence worker1 # [ 0.012349] pinctrl core: initialized pinctrl subsystem client # [ 0.005405] CPU features: detected: Data cache clean to Point of Persistence worker1 # [ 0.013226] DMI not present or invalid. worker2 # [ 0.005271] CPU features: detected: Data cache clean to the PoU not required for I/D coherence client # [ 0.005408] CPU features: detected: Data independent timing control (DIT) worker1 # [ 0.016601] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker2 # [ 0.005275] CPU features: detected: Common not Private translations lb # [ 0.005274] CPU features: detected: Data cache clean to Point of Persistence client # [ 0.005411] CPU features: detected: E0PD worker2 # [ 0.005278] CPU features: detected: CRC32 instructions worker1 # [ 0.018883] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations lb # [ 0.005277] CPU features: detected: Data independent timing control (DIT) client # [ 0.005414] CPU features: detected: Enhanced Counter Virtualization lb # [ 0.005280] CPU features: detected: E0PD worker2 # [ 0.005281] CPU features: detected: Data cache clean to Point of Deep Persistence worker1 # [ 0.019080] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations client # [ 0.005417] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) lb # [ 0.005283] CPU features: detected: Enhanced Counter Virtualization client # [ 0.005421] CPU features: detected: Enhanced Virtualization Traps worker1 # [ 0.019376] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations lb # [ 0.005286] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) worker2 # [ 0.005285] CPU features: detected: Data cache clean to Point of Persistence client # [ 0.005424] CPU features: detected: Fine Grained Traps worker1 # [ 0.019398] audit: initializing netlink subsys (disabled) lb # [ 0.005290] CPU features: detected: Enhanced Virtualization Traps worker2 # [ 0.005288] CPU features: detected: Data independent timing control (DIT) worker1 # [ 0.019957] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.005428] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker2 # [ 0.005292] CPU features: detected: E0PD lb # [ 0.005292] CPU features: detected: Fine Grained Traps worker1 # [ 0.019959] thermal_sys: Registered thermal governor 'bang_bang' client # [ 0.005433] CPU features: detected: RCpc load-acquire (LDAPR) worker2 # [ 0.005294] CPU features: detected: Enhanced Counter Virtualization lb # [ 0.005295] CPU features: detected: Generic authentication (architected QARMA5 algorithm) client # [ 0.005436] CPU features: detected: LSE atomic instructions worker1 # [ 0.019963] thermal_sys: Registered thermal governor 'step_wise' worker2 # [ 0.005297] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) lb # [ 0.005300] CPU features: detected: RCpc load-acquire (LDAPR) client # [ 0.005438] CPU features: detected: Privileged Access Never worker1 # [ 0.019965] thermal_sys: Registered thermal governor 'user_space' client # [ 0.005441] CPU features: detected: PMUv3 lb # [ 0.005303] CPU features: detected: LSE atomic instructions worker2 # [ 0.005301] CPU features: detected: Enhanced Virtualization Traps worker1 # [ 0.019971] thermal_sys: Registered thermal governor 'power_allocator' client # [ 0.005444] CPU features: detected: RAS Extension Support worker2 # [ 0.005304] CPU features: detected: Fine Grained Traps lb # [ 0.005306] CPU features: detected: Privileged Access Never lb # [ 0.005309] CPU features: detected: PMUv3 worker1 # [ 0.020004] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 client # [ 0.005447] CPU features: detected: RASv1p1 Extension Support worker2 # [ 0.005307] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker1 # [ 0.020012] cpuidle: using governor ladder lb # [ 0.005311] CPU features: detected: RAS Extension Support client # [ 0.005450] CPU features: detected: Random Number Generator worker1 # [ 0.020018] cpuidle: using governor menu worker2 # [ 0.005312] CPU features: detected: RCpc load-acquire (LDAPR) lb # [ 0.005314] CPU features: detected: RASv1p1 Extension Support client # [ 0.005452] CPU features: detected: Speculation barrier (SB) worker2 # [ 0.005315] CPU features: detected: LSE atomic instructions worker1 # [ 0.020211] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. lb # [ 0.005317] CPU features: detected: Random Number Generator client # [ 0.005455] CPU features: detected: Stage-2 Force Write-Back worker2 # [ 0.005318] CPU features: detected: Privileged Access Never worker1 # [ 0.020227] ASID allocator initialised with 65536 entries lb # [ 0.005319] CPU features: detected: Speculation barrier (SB) client # [ 0.005458] CPU features: detected: TLB range maintenance instructions worker2 # [ 0.005320] CPU features: detected: PMUv3 worker1 # [ 0.021478] Serial: AMBA PL011 UART driver lb # [ 0.005322] CPU features: detected: Stage-2 Force Write-Back worker2 # [ 0.005323] CPU features: detected: RAS Extension Support client # [ 0.005463] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker1 # [ 0.026781] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 lb # [ 0.005325] CPU features: detected: TLB range maintenance instructions worker2 # [ 0.005326] CPU features: detected: RASv1p1 Extension Support client # [ 0.005503] alternatives: applying system-wide alternatives worker1 # [ 0.026944] printk: console [ttyAMA0] enabled worker2 # [ 0.005328] CPU features: detected: Random Number Generator lb # [ 0.005329] CPU features: detected: Speculative Store Bypassing Safe (SSBS) client # [ 0.008604] CPU features: detected: BBM Level 2 without TLB conflict abort worker2 # [ 0.005331] CPU features: detected: Speculation barrier (SB) lb # [ 0.005366] alternatives: applying system-wide alternatives worker2 # [ 0.005334] CPU features: detected: Stage-2 Force Write-Back lb # [ 0.008450] CPU features: detected: BBM Level 2 without TLB conflict abort client # [ 0.008790] Memory: 1405640K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved) worker2 # [ 0.005336] CPU features: detected: TLB range maintenance instructions worker2 # [ 0.005341] CPU features: detected: Speculative Store Bypassing Safe (SSBS) client # [ 0.009212] devtmpfs: initialized lb # [ 0.008655] Memory: 1405680K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved) lb # [ 0.009047] devtmpfs: initialized worker2 # [ 0.005378] alternatives: applying system-wide alternatives client # [ 0.011062] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker2 # [ 0.008501] CPU features: detected: BBM Level 2 without TLB conflict abort lb # [ 0.010970] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) client # [ 0.011086] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). lb # [ 0.010994] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). client # [ 0.011282] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL worker2 # [ 0.008673] Memory: 1405676K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved) client # [ 0.011287] 0 pages in range for non-PLT usage lb # [ 0.011193] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL worker2 # [ 0.009089] devtmpfs: initialized worker1 # [ 0.151740] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages lb # [ 0.011197] 0 pages in range for non-PLT usage client # [ 0.011288] 508272 pages in range for PLT usage lb # [ 0.011199] 508272 pages in range for PLT usage client # [ 0.011387] pinctrl core: initialized pinctrl subsystem worker1 # [ 0.151761] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker2 # [ 0.010926] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) client # [ 0.012198] DMI not present or invalid. lb # [ 0.011317] pinctrl core: initialized pinctrl subsystem worker1 # [ 0.151766] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages worker2 # [ 0.010950] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). lb # [ 0.012120] DMI not present or invalid. client # [ 0.015451] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker1 # [ 0.151771] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page lb # [ 0.015313] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker2 # [ 0.011138] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL client # [ 0.018018] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker2 # [ 0.011142] 0 pages in range for non-PLT usage worker1 # [ 0.151776] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages lb # [ 0.017689] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker2 # [ 0.011143] 508272 pages in range for PLT usage client # [ 0.018216] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker1 # [ 0.151780] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.017884] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker2 # [ 0.011297] pinctrl core: initialized pinctrl subsystem client # [ 0.018428] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker2 # [ 0.012058] DMI not present or invalid. worker1 # [ 0.151786] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages lb # [ 0.018092] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations client # [ 0.018451] audit: initializing netlink subsys (disabled) worker2 # [ 0.015394] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker1 # [ 0.151790] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page lb # [ 0.018116] audit: initializing netlink subsys (disabled) client # [ 0.019024] thermal_sys: Registered thermal governor 'fair_share' worker2 # [ 0.017812] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations lb # [ 0.018753] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.019026] thermal_sys: Registered thermal governor 'bang_bang' worker2 # [ 0.018007] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations lb # [ 0.018755] thermal_sys: Registered thermal governor 'bang_bang' client # [ 0.019030] thermal_sys: Registered thermal governor 'step_wise' worker1 # [ 0.159541] fbcon: Taking over console worker1 # [ 0.159558] ACPI: Interpreter disabled. lb # [ 0.018759] thermal_sys: Registered thermal governor 'step_wise' worker2 # [ 0.018208] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations client # [ 0.019033] thermal_sys: Registered thermal governor 'user_space' lb # [ 0.018762] thermal_sys: Registered thermal governor 'user_space' worker2 # [ 0.018230] audit: initializing netlink subsys (disabled) client # [ 0.019038] thermal_sys: Registered thermal governor 'power_allocator' lb # [ 0.018767] thermal_sys: Registered thermal governor 'power_allocator' worker2 # [ 0.018809] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.019064] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 worker2 # [ 0.018811] thermal_sys: Registered thermal governor 'bang_bang' client # [ 0.019073] cpuidle: using governor ladder lb # [ 0.018802] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 client # [ 0.019079] cpuidle: using governor menu lb # [ 0.018811] cpuidle: using governor ladder worker2 # [ 0.018815] thermal_sys: Registered thermal governor 'step_wise' lb # [ 0.018816] cpuidle: using governor menu client # [ 0.019277] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker2 # [ 0.018818] thermal_sys: Registered thermal governor 'user_space' lb # [ 0.018998] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. client # [ 0.019294] ASID allocator initialised with 65536 entries worker2 # [ 0.018823] thermal_sys: Registered thermal governor 'power_allocator' client # [ 0.020542] Serial: AMBA PL011 UART driver lb # [ 0.019013] ASID allocator initialised with 65536 entries lb # [ 0.020257] Serial: AMBA PL011 UART driver worker2 # [ 0.018848] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 client # [ 0.025739] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 worker2 # [ 0.018857] cpuidle: using governor ladder worker2 # [ 0.018863] cpuidle: using governor menu client # [ 0.025910] printk: console [ttyAMA0] enabled lb # [ 0.025626] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 lb # [ 0.025796] printk: console [ttyAMA0] enabled worker2 # [ 0.019044] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker1 # [ 0.168596] iommu: Default domain type: Translated worker2 # [ 0.019059] ASID allocator initialised with 65536 entries worker2 # [ 0.020316] Serial: AMBA PL011 UART driver worker1 # [ 0.168608] iommu: DMA domain TLB invalidation policy: strict mode worker1 # [ 0.170363] SCSI subsystem initialized worker2 # [ 0.025493] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 worker2 # [ 0.025623] printk: console [ttyAMA0] enabled worker1 # [ 0.171068] usbcore: registered new interface driver usbfs worker1 # [ 0.171108] usbcore: registered new interface driver hub worker1 # [ 0.171124] usbcore: registered new device driver usb worker1 # [ 0.171468] pps_core: LinuxPPS API ver. 1 registered client # [ 0.150288] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker1 # [ 0.171475] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker1 # [ 0.171485] PTP clock support registered client # [ 0.150309] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker1 # [ 0.171534] EDAC MC: Ver: 3.0.0 lb # [ 0.150506] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages client # [ 0.150315] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages lb # [ 0.150524] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker1 # [ 0.176379] scmi_core: SCMI protocol bus registered client # [ 0.150319] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page lb # [ 0.150530] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages worker1 # [ 0.177376] FPGA manager framework client # [ 0.150324] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages lb # [ 0.150534] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page worker1 # [ 0.178348] vgaarb: loaded client # [ 0.150329] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.150547] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker1 # [ 0.178993] clocksource: Switched to clocksource arch_sys_counter client # [ 0.150333] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages lb # [ 0.150552] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page worker2 # [ 0.151304] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages client # [ 0.150338] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker2 # [ 0.151321] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page lb # [ 0.150556] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages worker2 # [ 0.151327] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages lb # [ 0.150561] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker2 # [ 0.151332] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page client # [ 0.158082] fbcon: Taking over console client # [ 0.158098] ACPI: Interpreter disabled. worker1 # [ 0.182914] VFS: Disk quotas dquot_6.6.0 worker2 # [ 0.151336] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages lb # [ 0.158412] fbcon: Taking over console worker1 # [ 0.182955] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) worker2 # [ 0.151341] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.158427] ACPI: Interpreter disabled. worker2 # [ 0.151346] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages worker2 # [ 0.151350] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker1 # [ 0.186896] netfs: FS-Cache loaded worker1 # [ 0.187644] pnp: PnP ACPI: disabled worker2 # [ 0.159173] fbcon: Taking over console worker2 # [ 0.159206] ACPI: Interpreter disabled. client # [ 0.167106] iommu: Default domain type: Translated client # [ 0.167117] iommu: DMA domain TLB invalidation policy: strict mode lb # [ 0.167035] iommu: Default domain type: Translated client # [ 0.167486] SCSI subsystem initialized lb # [ 0.167046] iommu: DMA domain TLB invalidation policy: strict mode lb # [ 0.167459] SCSI subsystem initialized client # [ 0.169568] usbcore: registered new interface driver usbfs client # [ 0.169600] usbcore: registered new interface driver hub lb # [ 0.169551] usbcore: registered new interface driver usbfs worker1 # [ 0.191096] NET: Registered PF_INET protocol family client # [ 0.169614] usbcore: registered new device driver usb lb # [ 0.169584] usbcore: registered new interface driver hub client # [ 0.169873] pps_core: LinuxPPS API ver. 1 registered worker1 # [ 0.191442] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) lb # [ 0.169600] usbcore: registered new device driver usb worker2 # [ 0.167828] iommu: Default domain type: Translated lb # [ 0.169916] pps_core: LinuxPPS API ver. 1 registered client # [ 0.169879] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker2 # [ 0.167838] iommu: DMA domain TLB invalidation policy: strict mode worker2 # [ 0.168205] SCSI subsystem initialized client # [ 0.169888] PTP clock support registered client # [ 0.169945] EDAC MC: Ver: 3.0.0 lb # [ 0.169923] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti lb # [ 0.169933] PTP clock support registered client # [ 0.174893] scmi_core: SCMI protocol bus registered lb # [ 0.169985] EDAC MC: Ver: 3.0.0 worker2 # [ 0.170292] usbcore: registered new interface driver usbfs worker2 # [ 0.170326] usbcore: registered new interface driver hub lb # [ 0.174985] scmi_core: SCMI protocol bus registered client # [ 0.175891] FPGA manager framework worker2 # [ 0.170342] usbcore: registered new device driver usb client # [ 0.176884] vgaarb: loaded worker2 # [ 0.170652] pps_core: LinuxPPS API ver. 1 registered lb # [ 0.176035] FPGA manager framework lb # [ 0.177045] vgaarb: loaded client # [ 0.177526] clocksource: Switched to clocksource arch_sys_counter worker2 # [ 0.170658] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker2 # [ 0.170669] PTP clock support registered lb # [ 0.177701] clocksource: Switched to clocksource arch_sys_counter worker2 # [ 0.170719] EDAC MC: Ver: 3.0.0 worker2 # [ 0.175637] scmi_core: SCMI protocol bus registered worker2 # [ 0.176655] FPGA manager framework client # [ 0.181466] VFS: Disk quotas dquot_6.6.0 worker2 # [ 0.177660] vgaarb: loaded client # [ 0.181507] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) lb # [ 0.181630] VFS: Disk quotas dquot_6.6.0 worker2 # [ 0.178318] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.181668] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) client # [ 0.185443] netfs: FS-Cache loaded client # [ 0.186049] pnp: PnP ACPI: disabled lb # [ 0.185662] netfs: FS-Cache loaded lb # [ 0.186280] pnp: PnP ACPI: disabled worker2 # [ 0.183289] VFS: Disk quotas dquot_6.6.0 worker2 # [ 0.183336] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) worker2 # [ 0.185031] netfs: FS-Cache loaded worker2 # [ 0.185156] pnp: PnP ACPI: disabled client # [ 0.189632] NET: Registered PF_INET protocol family client # [ 0.189900] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker2 # [ 0.189184] NET: Registered PF_INET protocol family lb # [ 0.189844] NET: Registered PF_INET protocol family worker2 # [ 0.189471] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) lb # [ 0.190114] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker1 # [ 0.221775] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) worker1 # [ 0.221826] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker1 # [ 0.221854] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) worker1 # [ 0.221926] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker1 # [ 0.222070] TCP: Hash tables configured (established 16384 bind 16384) worker1 # [ 0.222161] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker1 # [ 0.222197] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.222233] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.222326] NET: Registered PF_UNIX/PF_LOCAL protocol family worker1 # [ 0.222347] NET: Registered PF_XDP protocol family worker1 # [ 0.222364] PCI: CLS 0 bytes, default 64 worker1 # [ 0.222603] Trying to unpack rootfs image as initramfs... worker1 # [ 0.241216] kvm [1]: HYP mode not available client # [ 0.220501] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) client # [ 0.220556] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker2 # [ 0.220189] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) lb # [ 0.221972] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) client # [ 0.220585] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) worker2 # [ 0.220238] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) lb # [ 0.222021] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) client # [ 0.220660] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker2 # [ 0.220267] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) lb # [ 0.222050] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) client # [ 0.220806] TCP: Hash tables configured (established 16384 bind 16384) lb # [ 0.222129] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker2 # [ 0.220352] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) client # [ 0.220895] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker2 # [ 0.220495] TCP: Hash tables configured (established 16384 bind 16384) lb # [ 0.222272] TCP: Hash tables configured (established 16384 bind 16384) client # [ 0.220934] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.222362] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker2 # [ 0.220587] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) client # [ 0.220972] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.222400] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) worker2 # [ 0.220624] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.221067] NET: Registered PF_UNIX/PF_LOCAL protocol family lb # [ 0.222437] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) worker2 # [ 0.220661] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.221089] NET: Registered PF_XDP protocol family lb # [ 0.222554] NET: Registered PF_UNIX/PF_LOCAL protocol family worker2 # [ 0.220797] NET: Registered PF_UNIX/PF_LOCAL protocol family client # [ 0.221107] PCI: CLS 0 bytes, default 64 lb # [ 0.222582] NET: Registered PF_XDP protocol family worker2 # [ 0.220838] NET: Registered PF_XDP protocol family client # [ 0.221370] Trying to unpack rootfs image as initramfs... lb # [ 0.222601] PCI: CLS 0 bytes, default 64 worker2 # [ 0.220856] PCI: CLS 0 bytes, default 64 lb # [ 0.222868] Trying to unpack rootfs image as initramfs... worker2 # [ 0.221102] Trying to unpack rootfs image as initramfs... worker2 # [ 0.236527] kvm [1]: HYP mode not available client # [ 0.239820] kvm [1]: HYP mode not available lb # [ 0.239880] kvm [1]: HYP mode not available worker1 # [ 0.347667] Initialise system trusted keyrings worker1 # [ 0.348450] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker1 # [ 0.349759] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker1 # [ 0.350562] 9p: Installing v9fs 9p2000 file system support client # [ 0.350644] Initialise system trusted keyrings worker2 # [ 0.346918] Initialise system trusted keyrings client # [ 0.351501] workingset: timestamp_bits=42 max_order=19 bucket_order=0 lb # [ 0.351407] Initialise system trusted keyrings worker2 # [ 0.347713] workingset: timestamp_bits=42 max_order=19 bucket_order=0 client # [ 0.352779] squashfs: version 4.0 (2009/01/31) Phillip Lougher lb # [ 0.352197] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker2 # [ 0.349287] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker2 # [ 0.350097] 9p: Installing v9fs 9p2000 file system support lb # [ 0.353586] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker1 # [ 0.379447] Key type asymmetric registered worker1 # [ 0.379477] Asymmetric key parser 'x509' registered worker1 # [ 0.379556] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) client # [ 0.353606] 9p: Installing v9fs 9p2000 file system support worker1 # [ 0.381781] io scheduler mq-deadline registered worker1 # [ 0.381794] io scheduler kyber registered lb # [ 0.354451] 9p: Installing v9fs 9p2000 file system support worker1 # [ 0.395215] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker1 # [ 0.395946] ledtrig-cpu: registered to indicate activity on CPUs worker1 # [ 0.396356] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: client # [ 0.381880] Key type asymmetric registered worker1 # [ 0.396373] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 client # [ 0.381912] Asymmetric key parser 'x509' registered worker1 # [ 0.396386] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker2 # [ 0.378941] Key type asymmetric registered client # [ 0.381987] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) lb # [ 0.382667] Key type asymmetric registered worker2 # [ 0.378974] Asymmetric key parser 'x509' registered client # [ 0.384242] io scheduler mq-deadline registered lb # [ 0.382697] Asymmetric key parser 'x509' registered worker1 # [ 0.396394] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 client # [ 0.384254] io scheduler kyber registered worker2 # [ 0.379061] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) lb # [ 0.382773] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker1 # [ 0.396435] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker2 # [ 0.381274] io scheduler mq-deadline registered lb # [ 0.384967] io scheduler mq-deadline registered worker2 # [ 0.381286] io scheduler kyber registered lb # [ 0.384979] io scheduler kyber registered worker1 # [ 0.396463] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker1 # [ 0.396547] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker1 # [ 0.396557] pci_bus 0000:00: root bus resource [bus 00-ff] worker1 # [ 0.396563] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker1 # [ 0.396568] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.396573] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.396663] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.397123] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.397315] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.397332] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.393703] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker1 # [ 0.397363] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.397379] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.397850] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.395198] ledtrig-cpu: registered to indicate activity on CPUs worker1 # [ 0.398041] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] client # [ 0.395698] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker1 # [ 0.398057] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.398087] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.395718] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker1 # [ 0.398549] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint client # [ 0.395732] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker1 # [ 0.398736] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] worker1 # [ 0.398752] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.395742] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker1 # [ 0.398783] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.395765] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker1 # [ 0.399323] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.395792] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker1 # [ 0.399510] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] client # [ 0.395878] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker1 # [ 0.399526] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.395888] pci_bus 0000:00: root bus resource [bus 00-ff] worker1 # [ 0.399555] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.394710] pl061_gpio 9030000.pl061: PL061 GPIO chip registered client # [ 0.395895] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] lb # [ 0.398494] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker1 # [ 0.399574] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.395427] ledtrig-cpu: registered to indicate activity on CPUs lb # [ 0.399126] ledtrig-cpu: registered to indicate activity on CPUs client # [ 0.395900] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.400043] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker2 # [ 0.395850] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: lb # [ 0.399518] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: client # [ 0.395906] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.400242] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.395867] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 lb # [ 0.399534] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 client # [ 0.395966] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.400272] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.396423] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.399555] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker2 # [ 0.395880] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker1 # [ 0.400735] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint client # [ 0.396620] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.400926] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.399564] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker2 # [ 0.395889] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 client # [ 0.396640] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.400957] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.395912] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits lb # [ 0.399584] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits client # [ 0.396671] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.401363] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint client # [ 0.396688] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.399611] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker2 # [ 0.395936] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker1 # [ 0.401552] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] client # [ 0.397164] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.399688] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker2 # [ 0.396021] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker1 # [ 0.401819] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.397354] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.399698] pci_bus 0000:00: root bus resource [bus 00-ff] worker2 # [ 0.396031] pci_bus 0000:00: root bus resource [bus 00-ff] worker1 # [ 0.402014] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.397371] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.399703] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker2 # [ 0.396037] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker1 # [ 0.402045] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.397401] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.399709] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker2 # [ 0.396043] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.402506] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.399714] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker2 # [ 0.396048] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.402694] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.396113] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint lb # [ 0.399782] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.402724] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.396573] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.400230] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.396769] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.400422] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.396786] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.400439] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.396817] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.400470] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.396834] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.400486] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.397321] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.400952] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker2 # [ 0.397513] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.401143] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.397529] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.401159] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.397560] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.401189] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.398032] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint lb # [ 0.401661] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker2 # [ 0.398224] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] lb # [ 0.401868] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] worker2 # [ 0.398241] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.401884] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.398272] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.401915] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.398767] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.402398] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.398963] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.402668] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.398980] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.402689] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.399011] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.402720] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.426037] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker2 # [ 0.399033] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.402740] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.451386] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.426254] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] worker2 # [ 0.399631] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker1 # [ 0.451615] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.403231] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint client # [ 0.426273] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.399831] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.403425] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.451648] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.426305] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.399862] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.403457] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.452152] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint client # [ 0.426782] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.452441] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker2 # [ 0.400369] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint lb # [ 0.403923] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint client # [ 0.426975] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.452459] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.400564] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.404119] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.426992] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.452496] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.400595] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.427023] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.404149] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.427043] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.401002] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker1 # [ 0.453012] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.404583] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker1 # [ 0.453204] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.401194] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] lb # [ 0.404773] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] client # [ 0.427618] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker1 # [ 0.453221] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.427815] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.401457] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.405031] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.453251] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.401655] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.405224] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.427846] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.453875] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker2 # [ 0.401686] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.405255] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.428321] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker1 # [ 0.453886] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned client # [ 0.428517] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.402155] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.453892] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned client # [ 0.428548] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.453939] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned client # [ 0.429000] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint client # [ 0.429189] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker1 # [ 0.453988] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned client # [ 0.429443] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.454038] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned client # [ 0.429657] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.454091] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned client # [ 0.429688] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.454141] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned client # [ 0.430165] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.454191] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned client # [ 0.430361] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.454240] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned client # [ 0.430392] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.454288] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned client # [ 0.430869] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.431065] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.454336] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned client # [ 0.431096] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.454411] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned client # [ 0.431597] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker1 # [ 0.454459] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned client # [ 0.431876] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker1 # [ 0.454481] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker2 # [ 0.446427] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.431894] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.449817] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.454503] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker2 # [ 0.446477] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.431924] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.450042] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.454525] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned worker2 # [ 0.447007] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.432398] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.450075] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.454547] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker2 # [ 0.447279] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.432587] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.454569] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned lb # [ 0.450674] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.432604] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.447311] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.450869] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.454591] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned client # [ 0.432635] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.447799] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker1 # [ 0.454618] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned lb # [ 0.450900] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.448092] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] client # [ 0.433254] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker1 # [ 0.454641] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned lb # [ 0.451377] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker2 # [ 0.448110] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.433265] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned lb # [ 0.451661] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker1 # [ 0.454667] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned worker2 # [ 0.448141] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.433272] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned lb # [ 0.451678] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.454690] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker2 # [ 0.448615] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.451709] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.433319] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker1 # [ 0.454712] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned worker2 # [ 0.448806] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.454734] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned lb # [ 0.452179] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.433368] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker2 # [ 0.448822] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.452369] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.454755] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned client # [ 0.433416] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker2 # [ 0.448853] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.454777] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned lb # [ 0.452385] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.449442] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned client # [ 0.433464] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker1 # [ 0.454798] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned lb # [ 0.452416] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.449454] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker1 # [ 0.454819] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned client # [ 0.433513] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned lb # [ 0.453025] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker1 # [ 0.454842] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned worker2 # [ 0.449460] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned lb # [ 0.453037] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker1 # [ 0.454868] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] worker2 # [ 0.449507] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned lb # [ 0.453042] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker1 # [ 0.454878] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] worker2 # [ 0.449556] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker1 # [ 0.454882] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] lb # [ 0.453089] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker1 # [ 0.455767] pci 0000:00:07.0: enabling device (0000 -> 0002) worker2 # [ 0.449605] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned lb # [ 0.453138] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker2 # [ 0.449654] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned lb # [ 0.453187] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker2 # [ 0.449704] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned lb # [ 0.453238] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker2 # [ 0.449753] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned lb # [ 0.453287] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker2 # [ 0.449803] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned client # [ 0.473638] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned lb # [ 0.453337] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker2 # [ 0.449851] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned client # [ 0.473705] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned lb # [ 0.453385] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker2 # [ 0.449899] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned client # [ 0.473756] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned lb # [ 0.453434] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker2 # [ 0.449963] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned client # [ 0.473805] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned lb # [ 0.453481] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned worker2 # [ 0.450011] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned client # [ 0.473924] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned lb # [ 0.453550] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned worker2 # [ 0.450033] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned client # [ 0.473976] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned lb # [ 0.453597] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker2 # [ 0.450055] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned client # [ 0.474001] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned lb # [ 0.453619] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker2 # [ 0.450076] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned client # [ 0.474025] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned lb # [ 0.453640] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker2 # [ 0.450098] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned client # [ 0.474049] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned lb # [ 0.453662] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned worker2 # [ 0.450120] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned client # [ 0.474073] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned lb # [ 0.453684] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker2 # [ 0.450142] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned client # [ 0.474097] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned lb # [ 0.453723] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned worker2 # [ 0.450169] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned client # [ 0.474119] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned lb # [ 0.453745] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned worker2 # [ 0.450191] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned client # [ 0.474147] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned lb # [ 0.453772] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned worker2 # [ 0.450214] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned client # [ 0.474170] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned lb # [ 0.453794] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker2 # [ 0.450237] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned client # [ 0.474193] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned lb # [ 0.453817] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned worker2 # [ 0.450259] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned client # [ 0.474217] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned lb # [ 0.453840] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker2 # [ 0.450281] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned client # [ 0.474240] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned lb # [ 0.453863] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned worker2 # [ 0.450303] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned client # [ 0.474263] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned lb # [ 0.453885] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned worker2 # [ 0.450346] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned client # [ 0.474285] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned lb # [ 0.453912] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned worker2 # [ 0.450368] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned client # [ 0.474308] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned lb # [ 0.453935] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned worker2 # [ 0.450390] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned client # [ 0.474330] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned lb # [ 0.453957] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned worker2 # [ 0.450411] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned client # [ 0.474352] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned lb # [ 0.453979] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned worker2 # [ 0.450437] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] client # [ 0.474374] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned lb # [ 0.454001] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned worker2 # [ 0.450447] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] client # [ 0.474406] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.454029] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] worker2 # [ 0.450452] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] client # [ 0.474416] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] lb # [ 0.454038] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] worker2 # [ 0.451318] pci 0000:00:07.0: enabling device (0000 -> 0002) client # [ 0.474421] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] worker1 # [ 0.522065] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) lb # [ 0.454043] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] client # [ 0.475273] pci 0000:00:07.0: enabling device (0000 -> 0002) lb # [ 0.454898] pci 0000:00:07.0: enabling device (0000 -> 0002) worker1 # [ 0.525462] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker1 # [ 0.531218] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.534031] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) worker1 # [ 0.544449] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) client # [ 0.524171] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) lb # [ 0.524519] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) client # [ 0.527585] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker2 # [ 0.524987] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) lb # [ 0.531081] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker2 # [ 0.528163] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) lb # [ 0.533088] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker2 # [ 0.530195] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.555219] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) client # [ 0.537768] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.557092] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) client # [ 0.540126] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) worker1 # [ 0.558903] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker1 # [ 0.560974] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) worker1 # [ 0.562904] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker2 # [ 0.539757] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) lb # [ 0.543170] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) worker2 # [ 0.541856] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) lb # [ 0.545256] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) worker2 # [ 0.545663] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) worker1 # [ 0.576775] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) client # [ 0.550151] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) client # [ 0.552366] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) lb # [ 0.555463] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) worker2 # [ 0.552087] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) client # [ 0.554259] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) lb # [ 0.557305] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker2 # [ 0.554017] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) client # [ 0.556373] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) lb # [ 0.560906] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) client # [ 0.566555] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) client # [ 0.568489] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker2 # [ 0.563798] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) worker1 # [ 0.594125] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled lb # [ 0.567343] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) worker2 # [ 0.565623] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) lb # [ 0.569160] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker1 # [ 0.597046] msm_serial: driver initialized worker1 # [ 0.597234] SuperH (H)SCI(F) driver initialized worker1 # [ 0.597291] STM32 USART driver initialized client # [ 0.578988] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker2 # [ 0.575905] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) lb # [ 0.579518] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) client # [ 0.590789] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker2 # [ 0.587588] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled client # [ 0.593383] msm_serial: driver initialized worker2 # [ 0.590315] msm_serial: driver initialized lb # [ 0.595070] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled client # [ 0.594129] SuperH (H)SCI(F) driver initialized client # [ 0.594190] STM32 USART driver initialized worker2 # [ 0.591036] SuperH (H)SCI(F) driver initialized worker2 # [ 0.591093] STM32 USART driver initialized lb # [ 0.597753] msm_serial: driver initialized lb # [ 0.597931] SuperH (H)SCI(F) driver initialized lb # [ 0.597987] STM32 USART driver initialized worker1 # [ 0.630135] loop: module loaded worker1 # [ 0.630347] virtio_blk virtio2: 1/0/0 default/read/poll queues worker1 # [ 0.632528] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker1 # [ 0.643830] megasas: 07.734.00.00-rc1 client # [ 0.626607] loop: module loaded worker1 # [ 0.644661] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] client # [ 0.626811] virtio_blk virtio2: 1/0/0 default/read/poll queues worker2 # [ 0.623486] loop: module loaded worker1 # [ 0.646603] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.623679] virtio_blk virtio2: 1/0/0 default/read/poll queues client # [ 0.627922] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker1 # [ 0.646633] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.624485] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker1 # [ 0.648357] Using buffer write method worker1 # [ 0.648432] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] lb # [ 0.630340] loop: module loaded lb # [ 0.630732] virtio_blk virtio2: 1/0/0 default/read/poll queues worker1 # [ 0.650177] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.650204] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.631549] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker2 # [ 0.635043] megasas: 07.734.00.00-rc1 lb # [ 0.638411] megasas: 07.734.00.00-rc1 worker2 # [ 0.635910] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] lb # [ 0.639146] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] worker1 # [ 0.669108] Using buffer write method worker2 # [ 0.638030] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.669155] Concatenating MTD devices: worker2 # [ 0.638059] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.669159] (0): "0.flash" worker1 # [ 0.669163] (1): "0.flash" worker1 # [ 0.669174] into device "0.flash" client # [ 0.642258] megasas: 07.734.00.00-rc1 client # [ 0.643000] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] client # [ 0.645279] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 client # [ 0.645309] Intel/Sharp Extended Query Table at 0x0031 client # [ 0.647079] Using buffer write method worker2 # [ 0.647829] Using buffer write method client # [ 0.647155] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] lb # [ 0.651019] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.647920] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] client # [ 0.649051] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.651057] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.652751] Using buffer write method client # [ 0.649077] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.649542] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.649568] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.652836] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker2 # [ 0.663614] Using buffer write method client # [ 0.668346] Using buffer write method worker2 # [ 0.663655] Concatenating MTD devices: worker2 # [ 0.663659] (0): "0.flash" client # [ 0.668386] Concatenating MTD devices: worker2 # [ 0.663663] (1): "0.flash" client # [ 0.668391] (0): "0.flash" worker2 # [ 0.663672] into device "0.flash" client # [ 0.668395] (1): "0.flash" client # [ 0.668404] into device "0.flash" lb # [ 0.666502] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.666609] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.668673] Using buffer write method lb # [ 0.668712] Concatenating MTD devices: lb # [ 0.668716] (0): "0.flash" lb # [ 0.668721] (1): "0.flash" lb # [ 0.668732] into device "0.flash" worker1 # [ 0.956185] Freeing initrd memory: 26976K worker1 # [ 0.962436] tun: Universal TUN/TAP device driver, 1.6 worker1 # [ 0.966335] thunder_xcv, ver 1.0 worker1 # [ 0.966383] thunder_bgx, ver 1.0 worker1 # [ 0.966406] nicpf, ver 1.0 worker1 # [ 0.966993] e1000: Intel(R) PRO/1000 Network Driver worker1 # [ 0.968919] e1000: Copyright (c) 1999-2006 Intel Corporation. worker1 # [ 0.968956] e1000e: Intel(R) PRO/1000 Network Driver worker1 # [ 0.968965] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker1 # [ 0.968996] igb: Intel(R) Gigabit Ethernet Network Driver worker1 # [ 0.969002] igb: Copyright (c) 2007-2014 Intel Corporation. worker2 # [ 0.947507] Freeing initrd memory: 26944K worker1 # [ 0.969025] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker1 # [ 0.969031] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker1 # [ 0.969174] sky2: driver version 1.30 worker1 # [ 0.970805] usbcore: registered new interface driver usb-storage worker1 # [ 0.970858] usbcore: registered new interface driver usbserial_generic worker1 # [ 0.970873] usbserial: USB Serial support registered for generic worker1 # [ 0.971653] ehci-pci 0000:00:07.0: EHCI Host Controller client # [ 0.954514] Freeing initrd memory: 26972K worker1 # [ 0.971679] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker1 # [ 0.971844] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker2 # [ 0.953719] tun: Universal TUN/TAP device driver, 1.6 worker1 # [ 0.982725] hv_vmbus: registering driver hyperv_keyboard client # [ 0.960635] tun: Universal TUN/TAP device driver, 1.6 worker1 # [ 0.983052] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker1 # [ 0.983435] hub 1-0:1.0: USB hub found worker1 # [ 0.983456] hub 1-0:1.0: 6 ports detected worker2 # [ 0.957427] thunder_xcv, ver 1.0 worker2 # [ 0.957472] thunder_bgx, ver 1.0 worker2 # [ 0.957503] nicpf, ver 1.0 worker1 # [ 0.986853] rtc-pl031 9010000.pl031: registered as rtc0 worker2 # [ 0.958050] e1000: Intel(R) PRO/1000 Network Driver worker1 # [ 0.986883] rtc-pl031 9010000.pl031: setting system clock to 2026-09-25T14:59:21 UTC (1790348361) worker2 # [ 0.958057] e1000: Copyright (c) 1999-2006 Intel Corporation. worker2 # [ 0.958082] e1000e: Intel(R) PRO/1000 Network Driver client # [ 0.964457] thunder_xcv, ver 1.0 client # [ 0.964504] thunder_bgx, ver 1.0 worker1 # [ 0.989052] i2c_dev: i2c /dev entries driver worker2 # [ 0.958090] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. client # [ 0.964526] nicpf, ver 1.0 worker2 # [ 0.958122] igb: Intel(R) Gigabit Ethernet Network Driver client # [ 0.965084] e1000: Intel(R) PRO/1000 Network Driver lb # [ 0.966189] Freeing initrd memory: 26948K worker2 # [ 0.958128] igb: Copyright (c) 2007-2014 Intel Corporation. client # [ 0.965092] e1000: Copyright (c) 1999-2006 Intel Corporation. client # [ 0.965116] e1000e: Intel(R) PRO/1000 Network Driver worker2 # [ 0.958167] igbvf: Intel(R) Gigabit Virtual Function Network Driver client # [ 0.965124] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker2 # [ 0.958174] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker2 # [ 0.958309] sky2: driver version 1.30 client # [ 0.965155] igb: Intel(R) Gigabit Ethernet Network Driver client # [ 0.965160] igb: Copyright (c) 2007-2014 Intel Corporation. client # [ 0.965183] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker1 # [ 0.992917] sdhci: Secure Digital Host Controller Interface driver worker1 # [ 0.992930] sdhci: Copyright(c) Pierre Ossman client # [ 0.965189] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. client # [ 0.965322] sky2: driver version 1.30 worker1 # [ 0.993201] Synopsys Designware Multimedia Card Interface Driver worker1 # [ 0.993583] sdhci-pltfm: SDHCI platform and OF driver helper lb # [ 0.972440] tun: Universal TUN/TAP device driver, 1.6 worker2 # [ 0.967623] usbcore: registered new interface driver usb-storage worker2 # [ 0.967736] usbcore: registered new interface driver usbserial_generic worker1 # [ 0.997983] hid: raw HID events driver (C) Jiri Kosina worker2 # [ 0.967751] usbserial: USB Serial support registered for generic worker1 # [ 0.998246] usbcore: registered new interface driver usbhid worker1 # [ 0.998253] usbhid: USB HID core driver worker2 # [ 0.968349] hv_vmbus: registering driver hyperv_keyboard client # [ 0.974507] usbcore: registered new interface driver usb-storage worker2 # [ 0.969663] ehci-pci 0000:00:07.0: EHCI Host Controller client # [ 0.974561] usbcore: registered new interface driver usbserial_generic worker2 # [ 0.969690] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 lb # [ 0.976325] thunder_xcv, ver 1.0 client # [ 0.974574] usbserial: USB Serial support registered for generic lb # [ 0.976380] thunder_bgx, ver 1.0 worker2 # [ 0.969859] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 lb # [ 0.976404] nicpf, ver 1.0 client # [ 0.975159] hv_vmbus: registering driver hyperv_keyboard worker1 # [ 1.001338] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available lb # [ 0.976990] e1000: Intel(R) PRO/1000 Network Driver client # [ 0.976522] ehci-pci 0000:00:07.0: EHCI Host Controller worker1 # [ 1.002920] drop_monitor: Initializing network drop monitor service lb # [ 0.976999] e1000: Copyright (c) 1999-2006 Intel Corporation. client # [ 0.976553] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker2 # [ 0.975648] rtc-pl031 9010000.pl031: registered as rtc0 lb # [ 0.977023] e1000e: Intel(R) PRO/1000 Network Driver worker1 # [ 1.005024] NET: Registered PF_INET6 protocol family client # [ 0.976724] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 lb # [ 0.977032] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker2 # [ 0.975679] rtc-pl031 9010000.pl031: setting system clock to 2026-09-25T14:59:21 UTC (1790348361) worker1 # [ 1.006236] Segment Routing with IPv6 lb # [ 0.977062] igb: Intel(R) Gigabit Ethernet Network Driver worker2 # [ 0.976052] i2c_dev: i2c /dev entries driver worker1 # [ 1.006259] In-situ OAM (IOAM) with IPv6 lb # [ 0.977068] igb: Copyright (c) 2007-2014 Intel Corporation. worker1 # [ 1.006288] NET: Registered PF_PACKET protocol family client # [ 0.982534] rtc-pl031 9010000.pl031: registered as rtc0 worker2 # [ 0.979264] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 lb # [ 0.977091] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker2 # [ 0.979552] hub 1-0:1.0: USB hub found client # [ 0.982567] rtc-pl031 9010000.pl031: setting system clock to 2026-09-25T14:59:21 UTC (1790348361) lb # [ 0.977097] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker2 # [ 0.979633] hub 1-0:1.0: 6 ports detected client # [ 0.982910] i2c_dev: i2c /dev entries driver lb # [ 0.977235] sky2: driver version 1.30 client # [ 0.986110] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker1 # [ 1.008328] 9pnet: Installing 9P2000 support client # [ 0.986414] hub 1-0:1.0: USB hub found worker1 # [ 1.008375] Key type dns_resolver registered client # [ 0.986435] hub 1-0:1.0: 6 ports detected lb # [ 0.986512] usbcore: registered new interface driver usb-storage worker2 # [ 0.984173] sdhci: Secure Digital Host Controller Interface driver lb # [ 0.986726] usbcore: registered new interface driver usbserial_generic worker2 # [ 0.984186] sdhci: Copyright(c) Pierre Ossman lb # [ 0.986743] usbserial: USB Serial support registered for generic worker2 # [ 0.984449] Synopsys Designware Multimedia Card Interface Driver lb # [ 0.987343] hv_vmbus: registering driver hyperv_keyboard worker2 # [ 0.984827] sdhci-pltfm: SDHCI platform and OF driver helper lb # [ 0.988675] ehci-pci 0000:00:07.0: EHCI Host Controller client # [ 0.991008] sdhci: Secure Digital Host Controller Interface driver lb # [ 0.988705] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 client # [ 0.991023] sdhci: Copyright(c) Pierre Ossman worker2 # [ 0.989321] hid: raw HID events driver (C) Jiri Kosina lb # [ 0.988873] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 client # [ 0.991278] Synopsys Designware Multimedia Card Interface Driver worker2 # [ 0.989623] usbcore: registered new interface driver usbhid worker1 # [ 1.015176] registered taskstats version 1 client # [ 0.991664] sdhci-pltfm: SDHCI platform and OF driver helper worker2 # [ 0.989631] usbhid: USB HID core driver worker1 # [ 1.015392] Loading compiled-in X.509 certificates client # [ 0.993152] hid: raw HID events driver (C) Jiri Kosina lb # [ 0.994898] rtc-pl031 9010000.pl031: registered as rtc0 client # [ 0.993395] usbcore: registered new interface driver usbhid client # [ 0.993401] usbhid: USB HID core driver lb # [ 0.994940] rtc-pl031 9010000.pl031: setting system clock to 2026-09-25T14:59:21 UTC (1790348361) lb # [ 0.995303] i2c_dev: i2c /dev entries driver worker2 # [ 0.992779] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker2 # [ 0.994317] drop_monitor: Initializing network drop monitor service lb # [ 0.998483] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 lb # [ 0.998849] hub 1-0:1.0: USB hub found worker2 # [ 0.996431] NET: Registered PF_INET6 protocol family worker1 # [ 1.024241] Demotion targets for Node 0: null lb # [ 0.998870] hub 1-0:1.0: 6 ports detected worker1 # [ 1.024355] Key type .fscrypt registered worker2 # [ 0.997623] Segment Routing with IPv6 client # [ 0.999510] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker1 # [ 1.024362] Key type fscrypt-provisioning registered worker2 # [ 0.997656] In-situ OAM (IOAM) with IPv6 worker1 # [ 1.024468] ima: No TPM chip found, activating TPM-bypass! client # [ 1.001013] drop_monitor: Initializing network drop monitor service worker2 # [ 0.997685] NET: Registered PF_PACKET protocol family worker1 # [ 1.024488] ima: Allocated hash algorithm: sha1 client # [ 1.001203] NET: Registered PF_INET6 protocol family worker1 # [ 1.024510] ima: No architecture policies found client # [ 1.004523] Segment Routing with IPv6 client # [ 1.004544] In-situ OAM (IOAM) with IPv6 worker1 # [ 1.028786] input: gpio-keys as /devices/platform/gpio-keys/input/input0 client # [ 1.004572] NET: Registered PF_PACKET protocol family worker2 # [ 0.999550] 9pnet: Installing 9P2000 support lb # [ 1.003529] sdhci: Secure Digital Host Controller Interface driver worker2 # [ 0.999600] Key type dns_resolver registered lb # [ 1.003543] sdhci: Copyright(c) Pierre Ossman lb # [ 1.003809] Synopsys Designware Multimedia Card Interface Driver lb # [ 1.004183] sdhci-pltfm: SDHCI platform and OF driver helper lb # [ 1.005705] hid: raw HID events driver (C) Jiri Kosina client # [ 1.006277] 9pnet: Installing 9P2000 support client # [ 1.006324] Key type dns_resolver registered lb # [ 1.009469] usbcore: registered new interface driver usbhid lb # [ 1.009476] usbhid: USB HID core driver client # [ 1.013255] registered taskstats version 1 client # [ 1.013416] Loading compiled-in X.509 certificates worker2 # [ 1.006495] registered taskstats version 1 lb # [ 1.011877] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker2 # [ 1.006670] Loading compiled-in X.509 certificates lb # [ 1.013466] drop_monitor: Initializing network drop monitor service lb # [ 1.013662] NET: Registered PF_INET6 protocol family lb # [ 1.016926] Segment Routing with IPv6 lb # [ 1.016948] In-situ OAM (IOAM) with IPv6 lb # [ 1.016976] NET: Registered PF_PACKET protocol family worker2 # [ 1.015873] Demotion targets for Node 0: null worker2 # [ 1.016004] Key type .fscrypt registered worker2 # [ 1.016011] Key type fscrypt-provisioning registered worker2 # [ 1.016109] ima: No TPM chip found, activating TPM-bypass! client # [ 1.022285] Demotion targets for Node 0: null lb # [ 1.018753] 9pnet: Installing 9P2000 support worker2 # [ 1.016133] ima: Allocated hash algorithm: sha1 client # [ 1.022397] Key type .fscrypt registered lb # [ 1.018803] Key type dns_resolver registered worker2 # [ 1.016155] ima: No architecture policies found client # [ 1.022404] Key type fscrypt-provisioning registered client # [ 1.022504] ima: No TPM chip found, activating TPM-bypass! client # [ 1.022523] ima: Allocated hash algorithm: sha1 worker2 # [ 1.020498] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker1 # [ 1.047366] clk: Disabling unused clocks client # [ 1.022545] ima: No architecture policies found worker1 # [ 1.047397] PM: genpd: Disabling unused power domains client # [ 1.027041] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker1 # [ 1.051662] Freeing unused kernel memory: 4736K worker1 # [ 1.051878] Run /init as init process lb # [ 1.025891] registered taskstats version 1 lb # [ 1.026044] Loading compiled-in X.509 certificates lb # [ 1.035143] Demotion targets for Node 0: null lb # [ 1.035263] Key type .fscrypt registered lb # [ 1.035272] Key type fscrypt-provisioning registered lb # [ 1.035377] ima: No TPM chip found, activating TPM-bypass! lb # [ 1.035397] ima: Allocated hash algorithm: sha1 lb # [ 1.035419] ima: No architecture policies found lb # [ 1.039763] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker1 # [ 1.069401] systemd[1]: Successfully made /usr/ read-only. worker2 # [ 1.039629] clk: Disabling unused clocks worker2 # [ 1.039663] PM: genpd: Disabling unused power domains client # [ 1.045976] clk: Disabling unused clocks client # [ 1.046009] PM: genpd: Disabling unused power domains worker2 # [ 1.043988] Freeing unused kernel memory: 4736K worker2 # [ 1.044184] Run /init as init process client # [ 1.050306] Freeing unused kernel memory: 4736K client # [ 1.050502] Run /init as init process lb # [ 1.058959] clk: Disabling unused clocks lb # [ 1.058999] PM: genpd: Disabling unused power domains lb # [ 1.063350] Freeing unused kernel memory: 4736K worker2 # [ 1.060736] systemd[1]: Successfully made /usr/ read-only. lb # [ 1.063600] Run /init as init process client # [ 1.067960] systemd[1]: Successfully made /usr/ read-only. lb # [ 1.080115] systemd[1]: Successfully made /usr/ read-only. worker1 # [ 1.231039] usb 1-1: new high-speed USB device number 2 using ehci-pci worker2 # [ 1.226371] usb 1-1: new high-speed USB device number 2 using ehci-pci client # [ 1.233579] usb 1-1: new high-speed USB device number 2 using ehci-pci lb # [ 1.245764] usb 1-1: new high-speed USB device number 2 using ehci-pci worker1 # [ 1.380886] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker2 # [ 1.376175] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker1 # [ 1.404136] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 1.383314] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker1 # [ 1.409908] systemd[1]: Detected virtualization qemu. worker1 # [ 1.410833] systemd[1]: Detected architecture arm64. worker1 # [ 1.411770] systemd[1]: Running in initrd. worker1 # [ 1.412999] systemd[1]: Initializing machine ID from random generator. worker1 # [ 1.414238] systemd[1]: Hostname set to . lb # [ 1.395538] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker2 # [ 1.395508] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 1.402035] systemd[1]: Detected virtualization qemu. worker2 # [ 1.403198] systemd[1]: Detected architecture arm64. client # [ 1.402700] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 1.404265] systemd[1]: Running in initrd. client # [ 1.409344] systemd[1]: Detected virtualization qemu. worker2 # [ 1.405702] systemd[1]: Initializing machine ID from random generator. client # [ 1.410590] systemd[1]: Detected architecture arm64. client # [ 1.411789] systemd[1]: Running in initrd. worker2 # [ 1.407514] systemd[1]: Hostname set to . client # [ 1.413209] systemd[1]: Initializing machine ID from random generator. client # [ 1.414761] systemd[1]: Hostname set to . lb # [ 1.414897] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 1.421377] systemd[1]: Detected virtualization qemu. lb # [ 1.422520] systemd[1]: Detected architecture arm64. lb # [ 1.423551] systemd[1]: Running in initrd. lb # [ 1.424977] systemd[1]: Initializing machine ID from random generator. lb # [ 1.426631] systemd[1]: Hostname set to . worker1 # [ 1.467209] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 client # [ 1.465744] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker2 # [ 1.462591] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 lb # [ 1.481927] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker1 # [ 1.587050] usb 1-2: new high-speed USB device number 3 using ehci-pci client # [ 1.585596] usb 1-2: new high-speed USB device number 3 using ehci-pci worker2 # [ 1.582376] usb 1-2: new high-speed USB device number 3 using ehci-pci lb # [ 1.601765] usb 1-2: new high-speed USB device number 3 using ehci-pci worker1 # [ 1.664271] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 1.672538] systemd[1]: bpf-restrict-fs: LSM BPF program attached client # [ 1.687661] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 1.688770] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 1.737528] systemd[1]: Queued start job for default target Initrd Default Target. worker1 # [ 1.740652] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker1 # [ 1.742914] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker1 # [ 1.749355] systemd[1]: Created slice Slice /system/modprobe. worker1 # [ 1.750673] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 1.752175] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker1 # [ 1.753183] systemd[1]: Reached target Path Units. worker1 # [ 1.753995] systemd[1]: Reached target Slice Units. worker1 # [ 1.754760] systemd[1]: Reached target Swaps. worker1 # [ 1.755530] systemd[1]: Reached target Timer Units. worker1 # [ 1.756495] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 1.757700] systemd[1]: Listening on Journal Socket (/dev/log). worker1 # [ 1.758763] systemd[1]: Listening on Journal Sockets. worker1 # [ 1.759830] systemd[1]: Listening on udev Control Socket. worker1 # [ 1.760834] systemd[1]: Listening on udev Kernel Socket. worker1 # [ 1.761683] systemd[1]: Reached target Socket Units. worker1 # [ 1.764024] systemd[1]: Starting Create List of Static Device Nodes... client # [ 1.739316] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker1 # [ 1.765130] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 1.741623] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker1 # [ 1.772506] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 1.746252] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.748866] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 client # [ 1.759423] systemd[1]: Queued start job for default target Initrd Default Target. worker1 # [ 1.783969] systemd[1]: Starting Journal Service... worker2 # [ 1.756728] systemd[1]: Queued start job for default target Initrd Default Target. lb # [ 1.760498] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 lb # [ 1.763084] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 client # [ 1.767770] systemd[1]: Created slice Slice /system/modprobe. client # [ 1.768863] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 1.767996] systemd[1]: Queued start job for default target Initrd Default Target. client # [ 1.770221] systemd[1]: Expecting device /dev/disk/by-label/nixos... client # [ 1.771173] systemd[1]: Reached target Path Units. client # [ 1.771923] systemd[1]: Reached target Slice Units. client # [ 1.772718] systemd[1]: Reached target Swaps. worker2 # [ 1.768844] systemd[1]: Created slice Slice /system/modprobe. client # [ 1.773401] systemd[1]: Reached target Timer Units. client # [ 1.774451] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 1.770065] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 1.775615] systemd[1]: Listening on Journal Socket (/dev/log). worker2 # [ 1.771568] systemd[1]: Expecting device /dev/disk/by-label/nixos... client # [ 1.776642] systemd[1]: Listening on Journal Sockets. worker2 # [ 1.772667] systemd[1]: Reached target Path Units. lb # [ 1.776279] systemd[1]: Created slice Slice /system/modprobe. client # [ 1.777593] systemd[1]: Listening on udev Control Socket. worker2 # [ 1.773525] systemd[1]: Reached target Slice Units. client # [ 1.778559] systemd[1]: Listening on udev Kernel Socket. worker2 # [ 1.774453] systemd[1]: Reached target Swaps. lb # [ 1.777586] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 1.779432] systemd[1]: Reached target Socket Units. worker2 # [ 1.775297] systemd[1]: Reached target Timer Units. lb # [ 1.779136] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker2 # [ 1.776332] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 1.780216] systemd[1]: Reached target Path Units. lb # [ 1.781055] systemd[1]: Reached target Slice Units. client # [ 1.781841] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 1.777628] systemd[1]: Listening on Journal Socket (/dev/log). lb # [ 1.781964] systemd[1]: Reached target Swaps. worker2 # [ 1.778820] systemd[1]: Listening on Journal Sockets. client # [ 1.782903] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 1.782803] systemd[1]: Reached target Timer Units. worker2 # [ 1.779857] systemd[1]: Listening on udev Control Socket. lb # [ 1.783852] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 1.780933] systemd[1]: Listening on udev Kernel Socket. worker2 # [ 1.781849] systemd[1]: Reached target Socket Units. lb # [ 1.785089] systemd[1]: Listening on Journal Socket (/dev/log). lb # [ 1.786286] systemd[1]: Listening on Journal Sockets. lb # [ 1.787343] systemd[1]: Listening on udev Control Socket. worker1 # [ 1.811314] systemd[1]: Starting Load Kernel Modules... lb # [ 1.788422] systemd[1]: Listening on udev Kernel Socket. worker1 # [ 1.811428] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 1.789321] systemd[1]: Reached target Socket Units. worker2 # [ 1.784468] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 1.815437] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 1.784556] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 1.792097] systemd[1]: Mounting Kernel Configuration File System... lb # [ 1.791914] systemd[1]: Starting Create List of Static Device Nodes... lb # [ 1.791997] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 1.794512] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 1.826496] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 1.801943] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 1.802610] systemd[1]: Starting Journal Service... client # [ 1.809743] systemd[1]: Starting Journal Service... lb # [ 1.810098] systemd[1]: Starting Journal Service... worker1 # [ 1.834371] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker1 # [ 1.848348] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev client # [ 1.825779] systemd[1]: Starting Load Kernel Modules... client # [ 1.826598] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 1.850784] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker1 # [ 1.852533] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 1.825752] systemd[1]: Starting Load Kernel Modules... worker2 # [ 1.826918] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 1.865589] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 lb # [ 1.842166] systemd[1]: Starting Load Kernel Modules... worker1 # [ 1.865846] [drm] features: -virgl +edid -resource_blob -host_visible worker1 # [ 1.865854] [drm] features: -context_init lb # [ 1.843194] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 1.866585] [drm] number of scanouts: 1 worker1 # [ 1.866603] [drm] number of cap sets: 0 client # [ 1.849708] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 1.846616] systemd[1]: Starting Coldplug All udev Devices... client # [ 1.853752] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 1.849929] systemd-journald[72]: Collecting audit messages is disabled. client # [ 1.855335] systemd[1]: Mounted Kernel Configuration File System. lb # [ 1.859573] systemd[1]: Starting Coldplug All udev Devices... client # [ 1.864262] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. lb # [ 1.871174] systemd-journald[72]: Collecting audit messages is disabled. client # [ 1.873826] systemd-journald[72]: Collecting audit messages is disabled. worker1 # [ 1.891397] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker1 # [ 1.891421] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 worker2 # [ 1.870517] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 1.871384] systemd[1]: Mounted Kernel Configuration File System. client # [ 1.878268] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 1.879583] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker1 # [ 1.906759] systemd-journald[72]: Collecting audit messages is disabled. worker1 # [ 1.906925] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 1.911487] systemd[1]: Starting Create Static Device Nodes in /dev... lb # [ 1.890098] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 1.891221] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 1.882777] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 1.894693] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 1.899007] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 client # [ 1.899265] [drm] features: -virgl +edid -resource_blob -host_visible client # [ 1.899276] [drm] features: -context_init client # [ 1.900053] [drm] number of scanouts: 1 client # [ 1.900072] [drm] number of cap sets: 0 worker1 # [ 1.923357] Console: switching to colour frame buffer device 160x50 lb # [ 1.906942] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker2 # [ 1.907513] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker1 # [ 1.939713] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device client # [ 1.918231] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 1.920884] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 1.956261] systemd[1]: Finished Load Kernel Modules. lb # [ 1.921803] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev lb # [ 1.927992] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker1 # [ 1.958514] systemd[1]: Starting Apply Kernel Variables... client # [ 1.929874] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic lb # [ 1.928233] [drm] features: -virgl +edid -resource_blob -host_visible lb # [ 1.928243] [drm] features: -context_init client # [ 1.929894] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 lb # [ 1.928990] [drm] number of scanouts: 1 lb # [ 1.929009] [drm] number of cap sets: 0 worker2 # [ 1.934425] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker2 # [ 1.940787] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker2 # [ 1.941031] [drm] features: -virgl +edid -resource_blob -host_visible worker2 # [ 1.941042] [drm] features: -context_init worker2 # [ 1.941763] [drm] number of scanouts: 1 worker2 # [ 1.941781] [drm] number of cap sets: 0 worker2 # [ 1.955053] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 1.958429] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 1.957783] systemd[1]: Starting Create Static Device Nodes in /dev... client # [ 1.958003] Console: switching to colour frame buffer device 160x50 client # [ 1.962222] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker1 # [ 1.987505] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 1.961205] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker1 # [ 1.987737] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 1.961230] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 worker1 # [ 1.987774] systemd[1]: Reached target Local File Systems. worker1 # [ 1.993061] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 1.968011] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 1.966868] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker2 # [ 1.966887] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 client # [ 1.975385] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 1.975807] systemd[1]: Reached target Preparation for Local File Systems. client # [ 1.975847] systemd[1]: Reached target Local File Systems. client # [ 1.981981] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 1.986012] Console: switching to colour frame buffer device 160x50 worker2 # [ 1.990159] Console: switching to colour frame buffer device 160x50 client # [ 1.997705] systemd[1]: Finished Load Kernel Modules. worker1 # [ 2.031392] systemd[1]: Finished Apply Kernel Variables. client # [ 2.005924] systemd[1]: Starting Apply Kernel Variables... lb # [ 2.010400] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker2 # [ 2.010798] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device lb # [ 2.013415] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 2.013849] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 2.013883] systemd[1]: Reached target Local File Systems. lb # [ 2.022223] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker2 # [ 2.019759] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 2.021793] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 2.023682] systemd[1]: Reached target Local File Systems. worker2 # [ 2.031040] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker1 # [ 2.048568] systemd-modules-load[73]: Inserted module 'dm_mod' worker1 # [ 2.068686] systemd[1]: Started Journal Service. worker2 # [ 2.024626] systemd-modules-load[74]: Inserted module 'dm_mod' worker2 # [ 2.025929] systemd-modules-load[74]: Module 'virtio_balloon' is built in worker2 # [ 2.027020] systemd-modules-load[74]: Module 'virtio_console' is built in client # [ 2.050191] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 2.056515] systemd-modules-load[73]: Module 'virtio_balloon' is built in worker2 # [ 2.046455] systemd[1]: Finished Load Kernel Modules. worker1 # [ 2.057673] systemd-modules-load[73]: Module 'virtio_console' is built in worker2 # [ 2.046724] systemd[1]: Started Journal Service. lb # [ 2.050010] systemd[1]: Finished Load Kernel Modules. worker1 # [ 2.058766] systemd-modules-load[73]: Inserted module 'virtio_gpu' worker2 # [ 2.032193] systemd-modules-load[74]: Inserted module 'virtio_gpu' worker1 # [ 2.059756] systemd-modules-load[73]: Module 'virtio_rng' is built in lb # [ 2.053396] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 2.036329] systemd-modules-load[74]: Module 'virtio_rng' is built in lb # [ 2.040496] systemd-modules-load[73]: Inserted module 'dm_mod' client # [ 2.057707] systemd[1]: Started Journal Service. lb # [ 2.041865] systemd-modules-load[73]: Module 'virtio_balloon' is built in client # [ 2.044431] systemd-modules-load[73]: Inserted module 'dm_mod' client # [ 2.045701] systemd-modules-load[73]: Module 'virtio_balloon' is built in worker1 # [ 2.069450] systemd[1]: Starting Create System Files and Directories... lb # [ 2.049221] systemd-modules-load[73]: Module 'virtio_console' is built in lb # [ 2.069911] systemd[1]: Started Journal Service. lb # [ 2.054882] systemd-modules-load[73]: Inserted module 'virtio_gpu' client # [ 2.056333] systemd-modules-load[73]: Module 'virtio_console' is built in worker2 # [ 2.054493] systemd[1]: Starting Apply Kernel Variables... client # [ 2.060273] systemd-modules-load[73]: Inserted module 'virtio_gpu' client # [ 2.061399] systemd-modules-load[73]: Module 'virtio_rng' is built in lb # [ 2.060296] systemd-modules-load[73]: Module 'virtio_rng' is built in worker2 # [ 2.056420] systemd[1]: Starting Create System Files and Directories... worker1 # [ 2.089213] systemd-udevd[80]: Using default interface naming scheme 'v261'. client # [ 2.069154] systemd[1]: Starting Create System Files and Directories... worker1 # [ 2.105379] systemd[1]: Finished Create System Files and Directories. client # [ 2.084214] systemd-udevd[79]: Using default interface naming scheme 'v261'. lb # [ 2.085127] systemd[1]: Starting Create System Files and Directories... worker1 # [ 2.118360] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 2.096743] systemd[1]: Finished Create System Files and Directories. lb # [ 2.100900] systemd[1]: Finished Apply Kernel Variables. client # [ 2.113908] systemd[1]: Finished Create System Files and Directories. worker2 # [ 2.109362] systemd[1]: Finished Apply Kernel Variables. lb # [ 2.116312] systemd-udevd[79]: Using default interface naming scheme 'v261'. lb # [ 2.127131] systemd[1]: Finished Create System Files and Directories. client # [ 2.129235] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 2.129311] systemd-udevd[78]: Using default interface naming scheme 'v261'. lb # [ 2.146711] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 2.176664] systemd[1]: Starting Virtual Console Setup... worker2 # [ 2.156484] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 2.180630] systemd[1]: Starting Virtual Console Setup... worker1 # [ 2.228579] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 2.232347] systemd[1]: Finished Virtual Console Setup. lb # [ 2.212185] systemd[1]: Starting Virtual Console Setup... worker2 # [ 2.217408] systemd[1]: Starting Virtual Console Setup... client # [ 2.242256] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 2.244101] systemd[1]: Finished Virtual Console Setup. lb # [ 2.260588] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 2.264065] systemd[1]: Finished Virtual Console Setup. worker2 # [ 2.272549] systemd-vconsole-setup[100]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 2.276370] systemd[1]: Finished Virtual Console Setup. worker1 # [ 2.854755] systemd[1]: Finished Coldplug All udev Devices. worker1 # [ 2.861175] systemd[1]: Reached target System Initialization. worker1 # [ 2.862509] systemd[1]: Reached target Basic System. worker2 # [ 2.870706] systemd[1]: Finished Coldplug All udev Devices. worker2 # [ 2.872143] systemd[1]: Reached target System Initialization. worker2 # [ 2.876556] systemd[1]: Reached target Basic System. lb # [ 2.896597] systemd[1]: Finished Coldplug All udev Devices. lb # [ 2.898397] systemd[1]: Reached target System Initialization. lb # [ 2.899293] systemd[1]: Reached target Basic System. client # [ 2.907025] systemd[1]: Finished Coldplug All udev Devices. client # [ 2.908075] systemd[1]: Reached target System Initialization. client # [ 2.908978] systemd[1]: Reached target Basic System. worker1 # [ 3.008138] (udev-worker)[101]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 3.004182] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.045540] (udev-worker)[101]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.060458] (udev-worker)[106]: Network interface NamePolicy= disabled on kernel command line. client # [ 3.056477] (udev-worker)[99]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 3.056760] (udev-worker)[99]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 3.064469] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line. lb # [ 3.071102] (udev-worker)[96]: Network interface NamePolicy= disabled on kernel command line. lb # [ 3.076879] (udev-worker)[100]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. lb # [ 3.084217] (udev-worker)[100]: Network interface NamePolicy= disabled on kernel command line. client # [ 3.087113] (udev-worker)[90]: Network interface NamePolicy= disabled on kernel command line. client # [ 3.101983] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.149483] systemd[1]: Found device /dev/disk/by-label/nixos. worker1 # [ 3.151393] systemd[1]: Reached target Initrd Root Device. worker1 # [ 3.156771] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker2 # [ 3.154280] systemd[1]: Found device /dev/disk/by-label/nixos. worker2 # [ 3.157934] systemd[1]: Reached target Initrd Root Device. worker2 # [ 3.161480] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... lb # [ 3.176129] systemd[1]: Found device /dev/disk/by-label/nixos. client # [ 3.177249] systemd[1]: Found device /dev/disk/by-label/nixos. lb # [ 3.177302] systemd[1]: Reached target Initrd Root Device. worker1 # [ 3.202298] systemd-fsck[111]: nixos: clean, 12/65536 files, 13019/262144 blocks lb # [ 3.181365] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... client # [ 3.183100] systemd[1]: Reached target Initrd Root Device. client # [ 3.187141] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker1 # [ 3.210988] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker1 # [ 3.213951] systemd[1]: Mounting /sysroot... worker2 # [ 3.213492] systemd-fsck[109]: nixos: clean, 12/65536 files, 13019/262144 blocks worker2 # [ 3.223055] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker1 # [ 3.272465] EXT4-fs (vda): mounted filesystem d9b10c6c-1981-4594-a993-de61abb4c086 r/w with ordered data mode. Quota mode: none. worker2 # [ 3.229371] systemd[1]: Mounting /sysroot... worker1 # [ 3.259634] systemd[1]: Mounted /sysroot. lb # [ 3.235874] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks worker1 # [ 3.261531] systemd[1]: Reached target Initrd Root File System. worker1 # [ 3.268158] systemd[1]: Starting Mountpoints Configured in the Real Root... client # [ 3.246670] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks lb # [ 3.252237] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. client # [ 3.254913] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. lb # [ 3.254696] systemd[1]: Mounting /sysroot... client # [ 3.257670] systemd[1]: Mounting /sysroot... worker1 # [ 3.293743] systemd-sysroot-fstab-check[119]: /sysroot should be mounted in the initrd, will request daemon-reload. worker2 # [ 3.283929] EXT4-fs (vda): mounted filesystem 32fcf0a6-95eb-4478-af00-18bcbf63d3d3 r/w with ordered data mode. Quota mode: none. worker2 # [ 3.271082] systemd[1]: Mounted /sysroot. worker2 # [ 3.273213] systemd[1]: Reached target Initrd Root File System. worker1 # [ 3.299902] systemd[1]: Reload requested from client PID 119 ('systemd-sysroot') (unit initrd-parse-etc.service)... worker2 # [ 3.277497] systemd[1]: Starting Mountpoints Configured in the Real Root... worker1 # [ 3.305717] systemd[1]: Reloading... client # [ 3.313659] EXT4-fs (vda): mounted filesystem 80cc710e-c6b4-4276-acd8-4081619fb465 r/w with ordered data mode. Quota mode: none. lb # [ 3.312876] EXT4-fs (vda): mounted filesystem 80d3eab8-8922-4532-88a9-c338635f91f5 r/w with ordered data mode. Quota mode: none. client # [ 3.304825] systemd[1]: Mounted /sysroot. lb # [ 3.305463] systemd[1]: Mounted /sysroot. client # [ 3.307046] systemd[1]: Reached target Initrd Root File System. lb # [ 3.307790] systemd[1]: Reached target Initrd Root File System. client # [ 3.309758] systemd[1]: Starting Mountpoints Configured in the Real Root... worker2 # [ 3.304352] systemd-sysroot-fstab-check[117]: /sysroot should be mounted in the initrd, will request daemon-reload. lb # [ 3.316164] systemd[1]: Starting Mountpoints Configured in the Real Root... worker2 # [ 3.312600] systemd[1]: Reload requested from client PID 117 ('systemd-sysroot') (unit initrd-parse-etc.service)... worker2 # [ 3.316165] systemd[1]: Reloading... lb # [ 3.340354] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. client # [ 3.343648] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. client # [ 3.348209] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... lb # [ 3.348337] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... client # [ 3.353241] systemd[1]: Reloading... lb # [ 3.356159] systemd[1]: Reloading... worker1 # [ 3.528144] systemd[1]: Reloading finished in 223 ms. worker1 # [ 3.555858] systemd-sysroot-fstab-check[119]: Requesting initrd-fs.target/start/replace... worker1 # [ 3.561058] systemd-sysroot-fstab-check[119]: Requesting swap.target/start/replace... worker1 # [ 3.567464] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker1 # [ 3.570903] systemd[1]: Finished Mountpoints Configured in the Real Root. worker2 # [ 3.544914] systemd[1]: Reloading finished in 228 ms. worker1 # [ 3.573160] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. lb # [ 3.575130] systemd[1]: Reloading finished in 224 ms. client # [ 3.583660] systemd[1]: Reloading finished in 232 ms. worker2 # [ 3.578797] systemd-sysroot-fstab-check[117]: Requesting initrd-fs.target/start/replace... worker2 # [ 3.583579] systemd-sysroot-fstab-check[117]: Requesting swap.target/start/replace... worker2 # [ 3.591300] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker2 # [ 3.594589] systemd[1]: Finished Mountpoints Configured in the Real Root. worker2 # [ 3.596752] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. lb # [ 3.606476] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... lb # [ 3.611148] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... client # [ 3.617212] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... lb # [ 3.618520] systemd[1]: initrd-parse-etc.service: Deactivated successfully. client # [ 3.621585] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... lb # [ 3.621850] systemd[1]: Finished Mountpoints Configured in the Real Root. lb # [ 3.623455] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. client # [ 3.626769] systemd[1]: initrd-parse-etc.service: Deactivated successfully. client # [ 3.630179] systemd[1]: Finished Mountpoints Configured in the Real Root. client # [ 3.633058] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker1 # [ 3.804237] systemd[1]: Mounting /sysroot/nix/.ro-store... worker1 # [ 3.815095] systemd[1]: Mounting /sysroot/nix/.rw-store... worker1 # [ 3.829417] systemd[1]: Mounting /sysroot/run... worker1 # [ 3.861390] systemd[1]: Mounting /sysroot/tmp/shared... client # [ 3.840556] systemd[1]: Mounting /sysroot/nix/.ro-store... worker1 # [ 3.868317] systemd[1]: Mounting /sysroot/tmp/xchg... lb # [ 3.848204] systemd[1]: Mounting /sysroot/nix/.ro-store... lb # [ 3.857218] systemd[1]: Mounting /sysroot/nix/.rw-store... worker2 # [ 3.854705] systemd[1]: Mounting /sysroot/nix/.ro-store... client # [ 3.860337] systemd[1]: Mounting /sysroot/nix/.rw-store... client # [ 3.865001] systemd[1]: Mounting /sysroot/run... worker2 # [ 3.872336] systemd[1]: Mounting /sysroot/nix/.rw-store... lb # [ 3.879301] systemd[1]: Mounting /sysroot/run... worker2 # [ 3.877149] systemd[1]: Mounting /sysroot/run... lb # [ 3.897949] systemd[1]: Mounting /sysroot/tmp/shared... worker1 # [ 3.922240] systemd[1]: Mounted /sysroot/nix/.rw-store. client # [ 3.900071] systemd[1]: Mounting /sysroot/tmp/shared... worker2 # [ 3.896258] systemd[1]: Mounting /sysroot/tmp/shared... client # [ 3.904341] systemd[1]: Mounting /sysroot/tmp/xchg... worker1 # [ 3.951823] fuse: init (API version 7.45) worker2 # [ 3.917070] systemd[1]: Mounting /sysroot/tmp/xchg... lb # [ 3.922286] systemd[1]: Mounting /sysroot/tmp/xchg... worker1 # [ 3.962062] virtiofs virtio6: discovered new tag: nix-store worker1 # [ 3.962903] virtiofs virtio6: virtio_fs_setup_dax: No cache capability worker1 # [ 3.977882] virtiofs virtio7: discovered new tag: shared worker1 # [ 3.978773] virtiofs virtio7: virtio_fs_setup_dax: No cache capability client # [ 3.945280] systemd[1]: Mounted /sysroot/nix/.rw-store. worker1 # [ 3.985904] virtiofs virtio8: discovered new tag: xchg worker1 # [ 3.986775] virtiofs virtio8: virtio_fs_setup_dax: No cache capability worker1 # [ 3.974882] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 3.977171] fuse: init (API version 7.45) worker1 # [ 3.985598] systemd[1]: Mounted /sysroot/run. lb # [ 3.969651] systemd[1]: Mounted /sysroot/nix/.rw-store. client # [ 3.972149] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 3.997466] virtiofs virtio6: discovered new tag: nix-store worker1 # [ 4.009178] systemd[1]: Mounted /sysroot/nix/.ro-store. worker2 # [ 3.984152] systemd[1]: Mounted /sysroot/nix/.rw-store. worker1 # [ 4.012560] systemd[1]: Mounted /sysroot/tmp/shared. client # [ 4.011555] fuse: init (API version 7.45) worker1 # [ 4.018054] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker1 # [ 4.021249] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.014092] fuse: init (API version 7.45) client # [ 4.018388] virtiofs virtio6: discovered new tag: nix-store worker1 # [ 4.025664] systemd[1]: Mounted /sysroot/tmp/xchg. client # [ 4.019284] virtiofs virtio6: virtio_fs_setup_dax: No cache capability worker2 # [ 4.007408] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 4.013824] systemd[1]: Mounted /sysroot/run. worker2 # [ 4.010788] systemd[1]: Mounted /sysroot/run. client # [ 4.035378] virtiofs virtio7: discovered new tag: shared client # [ 4.036321] virtiofs virtio7: virtio_fs_setup_dax: No cache capability worker2 # [ 4.033616] virtiofs virtio6: discovered new tag: nix-store client # [ 4.043314] virtiofs virtio8: discovered new tag: xchg client # [ 4.044260] virtiofs virtio8: virtio_fs_setup_dax: No cache capability worker2 # [ 4.043484] virtiofs virtio6: virtio_fs_setup_dax: No cache capability lb # [ 4.049524] virtiofs virtio6: virtio_fs_setup_dax: No cache capability lb # [ 4.036397] systemd[1]: Mounted /sysroot/run. client # [ 4.046793] systemd[1]: Mounted /sysroot/tmp/shared. lb # [ 4.047226] systemd[1]: Starting rw-sysroot-nix-store.service... worker2 # [ 4.060491] virtiofs virtio7: discovered new tag: shared worker2 # [ 4.061351] virtiofs virtio7: virtio_fs_setup_dax: No cache capability client # [ 4.053734] systemd[1]: Mounted /sysroot/nix/.ro-store. client # [ 4.060296] systemd[1]: Mounted /sysroot/tmp/xchg. worker2 # [ 4.072914] virtiofs virtio8: discovered new tag: xchg worker2 # [ 4.073831] virtiofs virtio8: virtio_fs_setup_dax: No cache capability worker2 # [ 4.059971] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 4.065491] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 4.062650] systemd[1]: Finished rw-sysroot-nix-store.service. client # [ 4.068855] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 4.085044] virtiofs virtio7: discovered new tag: shared worker2 # [ 4.079435] systemd[1]: Mounted /sysroot/tmp/shared. worker2 # [ 4.082621] systemd[1]: Mounted /sysroot/nix/.ro-store. worker2 # [ 4.086486] systemd[1]: Mounted /sysroot/tmp/xchg. lb # [ 4.107856] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 4.120670] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 4.145256] virtiofs virtio7: virtio_fs_setup_dax: No cache capability lb # [ 4.162858] virtiofs virtio8: discovered new tag: xchg lb # [ 4.199743] virtiofs virtio8: virtio_fs_setup_dax: No cache capability lb # [ 4.258668] systemd[1]: Mounted /sysroot/tmp/shared. lb # [ 4.278975] systemd[1]: Mounted /sysroot/tmp/xchg. lb # [ 4.290078] systemd[1]: Mounted /sysroot/nix/.ro-store. worker1 # [ 4.506239] (udev-worker)[93]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker1 # [ 4.514352] (udev-worker)[93]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory worker1 # [ 4.550314] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.552727] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 4.556329] systemd[1]: Stopping Virtual Console Setup... worker1 # [ 4.561090] systemd[1]: Starting Virtual Console Setup... worker1 # [ 4.569529] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.571326] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 4.574763] systemd[1]: Starting Virtual Console Setup... worker2 # [ 4.549699] (udev-worker)[89]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker2 # [ 4.557203] (udev-worker)[89]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory client # [ 4.564793] (udev-worker)[94]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory client # [ 4.571267] (udev-worker)[94]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory worker1 # [ 4.606994] systemd-vconsole-setup[154]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 4.610887] systemd[1]: Finished Virtual Console Setup. worker2 # [ 4.590114] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 4.592393] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 4.595415] systemd[1]: Stopping Virtual Console Setup... worker2 # [ 4.600395] systemd[1]: Starting Virtual Console Setup... client # [ 4.609586] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.611929] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 4.607504] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.610834] (udev-worker)[95]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker2 # [ 4.609277] systemd[1]: Stopped Virtual Console Setup. client # [ 4.616441] systemd[1]: Stopping Virtual Console Setup... client # [ 4.617298] systemd[1]: Starting Virtual Console Setup... worker2 # [ 4.616146] systemd[1]: Starting Virtual Console Setup... lb # [ 4.618510] (udev-worker)[95]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory client # [ 4.625802] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.629886] systemd[1]: Stopped Virtual Console Setup. client # [ 4.634066] systemd[1]: Starting Virtual Console Setup... lb # [ 4.651071] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 4.647860] systemd-vconsole-setup[154]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 4.651776] systemd[1]: Finished Virtual Console Setup. lb # [ 4.657833] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.658656] systemd[1]: Stopping Virtual Console Setup... lb # [ 4.664161] systemd[1]: Starting Virtual Console Setup... client # [ 4.665460] systemd-vconsole-setup[154]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 4.669540] systemd[1]: Finished Virtual Console Setup. lb # [ 4.678462] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.679691] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.681146] systemd[1]: Starting Virtual Console Setup... lb # [ 4.704702] systemd-vconsole-setup[155]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 4.708414] systemd[1]: Finished Virtual Console Setup. worker1 # [ 4.810337] systemd[1]: Mounting /sysroot/nix/store... client # [ 4.844663] systemd[1]: Mounting /sysroot/nix/store... lb # [ 4.847805] systemd[1]: Mounting /sysroot/nix/store... worker1 # [ 4.873437] systemd[1]: Mounted /sysroot/nix/store. worker1 # [ 4.876163] systemd[1]: Reached target Initrd File Systems. worker1 # [ 4.879268] systemd[1]: Starting Find NixOS closure... worker2 # [ 4.852972] systemd[1]: Mounting /sysroot/nix/store... worker1 # [ 4.888467] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... client # [ 4.885698] systemd[1]: Mounted /sysroot/nix/store. client # [ 4.888236] systemd[1]: Reached target Initrd File Systems. client # [ 4.890903] systemd[1]: Starting Find NixOS closure... lb # [ 4.891022] systemd[1]: Mounted /sysroot/nix/store. lb # [ 4.894123] systemd[1]: Reached target Initrd File Systems. lb # [ 4.896086] systemd[1]: Starting Find NixOS closure... client # [ 4.900428] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker2 # [ 4.896617] systemd[1]: Mounted /sysroot/nix/store. lb # [ 4.902194] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker2 # [ 4.899055] systemd[1]: Reached target Initrd File Systems. worker1 # [ 4.929935] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker2 # [ 4.903171] systemd[1]: Starting Find NixOS closure... worker1 # [ 4.934678] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker2 # [ 4.912424] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker1 # [ 4.941429] systemd[1]: Finished Find NixOS closure. worker1 # [ 4.944301] systemd[1]: Reached target Initrd Default Target. worker1 # [ 4.946205] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... lb # [ 4.938969] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. client # [ 4.940712] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. lb # [ 4.940570] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. client # [ 4.946217] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker1 # [ 4.971898] systemd[1]: initrd-cleanup.service: Deactivated successfully. worker1 # [ 4.973637] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. lb # [ 4.950457] systemd[1]: Finished Find NixOS closure. lb # [ 4.953118] systemd[1]: Reached target Initrd Default Target. worker1 # [ 4.977117] systemd[1]: Stopped target Initrd Default Target. lb # [ 4.954491] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... client # [ 4.956624] systemd[1]: Finished Find NixOS closure. worker1 # [ 4.980248] systemd[1]: Stopped target Basic System. client # [ 4.958544] systemd[1]: Reached target Initrd Default Target. worker1 # [ 4.981115] systemd[1]: Stopped target Initrd Root Device. worker2 # [ 4.953826] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. client # [ 4.961874] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker1 # [ 4.984710] systemd[1]: Stopped target Path Units. worker2 # [ 4.957451] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker1 # [ 4.988745] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker1 # [ 4.990758] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker1 # [ 4.994745] systemd[1]: Stopped target Slice Units. worker2 # [ 4.967622] systemd[1]: Finished Find NixOS closure. worker1 # [ 4.996889] systemd[1]: Stopped target Socket Units. worker2 # [ 4.970371] systemd[1]: Reached target Initrd Default Target. worker2 # [ 4.971790] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... lb # [ 4.977766] systemd[1]: Stopped target Initrd Default Target. worker1 # [ 5.002234] systemd[1]: Stopped target System Initialization. lb # [ 4.979486] systemd[1]: Stopped target Basic System. worker1 # [ 5.003245] systemd[1]: Stopped target Swaps. worker1 # [ 5.004233] systemd[1]: Stopped target Timer Units. lb # [ 4.980733] systemd[1]: Stopped target Initrd Root Device. lb # [ 4.981955] systemd[1]: Stopped target Path Units. worker1 # [ 5.005617] systemd[1]: dbus.socket: Deactivated successfully. worker1 # [ 5.006508] systemd[1]: Closed D-Bus System Message Bus Socket. worker1 # [ 5.007401] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. lb # [ 4.984298] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. lb # [ 4.987623] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. lb # [ 4.989327] systemd[1]: Stopped target Slice Units. worker1 # [ 5.013881] systemd[1]: Stopped Find NixOS closure. client # [ 4.991446] systemd[1]: Stopped target Initrd Default Target. client # [ 4.993224] systemd[1]: Stopped target Basic System. lb # [ 4.992160] systemd[1]: Stopped target Socket Units. lb # [ 4.993135] systemd[1]: Stopped target System Initialization. client # [ 4.994396] systemd[1]: Stopped target Initrd Root Device. worker1 # [ 5.017387] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 4.995656] systemd[1]: Stopped target Path Units. lb # [ 4.994890] systemd[1]: Stopped target Swaps. lb # [ 4.996610] systemd[1]: Stopped target Timer Units. worker1 # [ 5.022025] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker1 # [ 5.023631] systemd[1]: Stopped Apply Kernel Variables. client # [ 5.001643] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. lb # [ 5.001442] systemd[1]: dbus.socket: Deactivated successfully. client # [ 5.003007] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker1 # [ 5.026024] systemd[1]: systemd-modules-load.service: Deactivated successfully. lb # [ 5.003976] systemd[1]: Closed D-Bus System Message Bus Socket. worker2 # [ 4.999739] systemd[1]: Stopped target Initrd Default Target. client # [ 5.005817] systemd[1]: Stopped target Slice Units. worker1 # [ 5.028424] systemd[1]: Stopped Load Kernel Modules. worker2 # [ 5.001792] systemd[1]: Stopped target Basic System. client # [ 5.008427] systemd[1]: Stopped target Socket Units. worker2 # [ 5.004342] systemd[1]: Stopped target Initrd Root Device. lb # [ 5.008616] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker1 # [ 5.032451] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker2 # [ 5.005494] systemd[1]: Stopped target Path Units. client # [ 5.010444] systemd[1]: Stopped target System Initialization. worker1 # [ 5.033682] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. lb # [ 5.010931] systemd[1]: Stopped Find NixOS closure. worker2 # [ 5.008332] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker1 # [ 5.035751] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker1 # [ 5.037556] systemd[1]: Stopped Create System Files and Directories. worker2 # [ 5.010422] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker1 # [ 5.038489] systemd[1]: Stopped target Local File Systems. lb # [ 5.015315] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 5.039298] systemd[1]: Stopped target Preparation for Local File Systems. client # [ 5.017274] systemd[1]: Stopped target Swaps. worker1 # [ 5.040490] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. lb # [ 5.017145] systemd[1]: systemd-sysctl.service: Deactivated successfully. client # [ 5.019082] systemd[1]: Stopped target Timer Units. worker1 # [ 5.041505] systemd[1]: Stopped Coldplug All udev Devices. lb # [ 5.018134] systemd[1]: Stopped Apply Kernel Variables. lb # [ 5.018896] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker1 # [ 5.042292] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker2 # [ 5.015097] systemd[1]: Stopped target Slice Units. client # [ 5.020827] systemd[1]: dbus.socket: Deactivated successfully. lb # [ 5.019918] systemd[1]: Stopped Load Kernel Modules. worker1 # [ 5.043337] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 5.022853] systemd[1]: Closed D-Bus System Message Bus Socket. worker2 # [ 5.020772] systemd[1]: Stopped target Socket Units. worker1 # [ 5.048415] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 5.022632] systemd[1]: Stopped target System Initialization. lb # [ 5.026694] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. client # [ 5.029488] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker1 # [ 5.052163] systemd[1]: systemd-udevd.service: Deactivated successfully. worker2 # [ 5.025262] systemd[1]: Stopped target Swaps. lb # [ 5.027989] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker1 # [ 5.053364] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker2 # [ 5.026851] systemd[1]: Stopped target Timer Units. lb # [ 5.030630] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. client # [ 5.032410] systemd[1]: Stopped Find NixOS closure. lb # [ 5.031706] systemd[1]: Stopped Create System Files and Directories. worker2 # [ 5.029194] systemd[1]: dbus.socket: Deactivated successfully. worker1 # [ 5.056186] systemd[1]: systemd-udevd.service: Consumed 1.491s CPU time over 3.042s wall clock time, 25.4M memory peak. client # [ 5.035083] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 5.060259] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. client # [ 5.038228] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker1 # [ 5.061348] systemd[1]: Closed udev Control Socket. worker1 # [ 5.062075] systemd[1]: Starting Cleanup udev Database... worker2 # [ 5.035635] systemd[1]: Closed D-Bus System Message Bus Socket. worker1 # [ 5.064111] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker2 # [ 5.038442] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. lb # [ 5.042776] systemd[1]: Stopped target Local File Systems. worker2 # [ 5.039618] systemd[1]: Stopped Find NixOS closure. lb # [ 5.043664] systemd[1]: Stopped target Preparation for Local File Systems. client # [ 5.045729] systemd[1]: Stopped Apply Kernel Variables. worker1 # [ 5.068398] systemd[1]: Stopped Create Static Device Nodes in /dev. worker1 # [ 5.069332] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker1 # [ 5.070467] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. client # [ 5.048281] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker2 # [ 5.044261] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 5.048982] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. lb # [ 5.050106] systemd[1]: Stopped Coldplug All udev Devices. worker2 # [ 5.046287] systemd[1]: systemd-sysctl.service: Deactivated successfully. lb # [ 5.050909] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker2 # [ 5.047330] systemd[1]: Stopped Apply Kernel Variables. client # [ 5.052534] systemd[1]: Stopped Load Kernel Modules. lb # [ 5.051964] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 5.076156] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker1 # [ 5.077283] systemd[1]: Stopped Create List of Static Device Nodes. worker1 # [ 5.078176] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 5.055364] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 5.079190] systemd[1]: Finished rw-sysroot-nix-store.service. client # [ 5.057698] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. lb # [ 5.058262] systemd[1]: initrd-cleanup.service: Deactivated successfully. client # [ 5.059138] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker2 # [ 5.054701] systemd[1]: systemd-modules-load.service: Deactivated successfully. lb # [ 5.059356] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker2 # [ 5.055804] systemd[1]: Stopped Load Kernel Modules. lb # [ 5.060657] systemd[1]: systemd-udevd.service: Deactivated successfully. lb # [ 5.061641] systemd[1]: Stopped Rule-based Manager for Device Events and Files. client # [ 5.063671] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. lb # [ 5.062661] systemd[1]: systemd-udevd.service: Consumed 1.533s CPU time over 3.032s wall clock time, 22.2M memory peak. client # [ 5.065093] systemd[1]: Stopped Create System Files and Directories. worker2 # [ 5.061234] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker2 # [ 5.062390] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. client # [ 5.068386] systemd[1]: Stopped target Local File Systems. worker2 # [ 5.063485] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. client # [ 5.069314] systemd[1]: Stopped target Preparation for Local File Systems. lb # [ 5.069176] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. client # [ 5.071682] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. lb # [ 5.072268] systemd[1]: Closed udev Control Socket. worker2 # [ 5.068432] systemd[1]: Stopped Create System Files and Directories. client # [ 5.076243] systemd[1]: Stopped Coldplug All udev Devices. lb # [ 5.076208] systemd[1]: Starting Cleanup udev Database... client # [ 5.077077] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker1 # [ 5.099268] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker2 # [ 5.072261] systemd[1]: Stopped target Local File Systems. worker2 # [ 5.073144] systemd[1]: Stopped target Preparation for Local File Systems. client # [ 5.078155] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 5.077099] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. client # [ 5.079211] systemd[1]: Stopped Virtual Console Setup. lb # [ 5.078239] systemd[1]: Stopped Create Static Device Nodes in /dev. worker2 # [ 5.074960] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. client # [ 5.079978] systemd[1]: initrd-cleanup.service: Deactivated successfully. worker1 # [ 5.103234] systemd[1]: Finished Cleanup udev Database. lb # [ 5.080187] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. client # [ 5.081657] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker1 # [ 5.104530] systemd[1]: Reached target Switch Root. worker2 # [ 5.077683] systemd[1]: Stopped Coldplug All udev Devices. client # [ 5.082697] systemd[1]: systemd-udevd.service: Deactivated successfully. worker2 # [ 5.078635] systemd[1]: Stopping Rule-based Manager for Device Events and Files... client # [ 5.083652] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker2 # [ 5.079730] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 5.084166] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. client # [ 5.085011] systemd[1]: systemd-udevd.service: Consumed 1.496s CPU time over 3.092s wall clock time, 22.1M memory peak. worker1 # [ 5.108422] systemd[1]: Starting NixOS Activation... worker2 # [ 5.081087] systemd[1]: Stopped Virtual Console Setup. lb # [ 5.085269] systemd[1]: kmod-static-nodes.service: Deactivated successfully. client # [ 5.086518] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. worker2 # [ 5.081856] systemd[1]: initrd-cleanup.service: Deactivated successfully. client # [ 5.087554] systemd[1]: Closed udev Control Socket. worker2 # [ 5.082837] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker2 # [ 5.083779] systemd[1]: systemd-udevd.service: Deactivated successfully. lb # [ 5.088175] systemd[1]: Stopped Create List of Static Device Nodes. lb # [ 5.089251] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 5.092191] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 5.088380] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker2 # [ 5.090579] systemd[1]: systemd-udevd.service: Consumed 1.502s CPU time over 3.040s wall clock time, 22.3M memory peak. client # [ 5.096377] systemd[1]: Starting Cleanup udev Database... client # [ 5.097253] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. client # [ 5.098750] systemd[1]: Stopped Create Static Device Nodes in /dev. worker2 # [ 5.096651] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. worker2 # [ 5.097938] systemd[1]: Closed udev Control Socket. worker2 # [ 5.098692] systemd[1]: Starting Cleanup udev Database... client # [ 5.104176] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker2 # [ 5.100134] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. client # [ 5.105396] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. client # [ 5.106506] systemd[1]: kmod-static-nodes.service: Deactivated successfully. client # [ 5.107494] systemd[1]: Stopped Create List of Static Device Nodes. worker2 # [ 5.104220] systemd[1]: Stopped Create Static Device Nodes in /dev. worker2 # [ 5.105167] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker2 # [ 5.106298] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. client # [ 5.112552] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 5.113584] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 5.116187] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker2 # [ 5.112418] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker2 # [ 5.113513] systemd[1]: Stopped Create List of Static Device Nodes. worker2 # [ 5.114403] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 5.119265] systemd[1]: Finished Cleanup udev Database. worker2 # [ 5.115417] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 5.120679] systemd[1]: Reached target Switch Root. lb # [ 5.124163] systemd[1]: Starting NixOS Activation... client # [ 5.139727] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker2 # [ 5.138388] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. client # [ 5.144679] systemd[1]: Finished Cleanup udev Database. client # [ 5.145513] systemd[1]: Reached target Switch Root. worker2 # [ 5.140548] systemd[1]: Finished Cleanup udev Database. client # [ 5.146357] systemd[1]: Starting NixOS Activation... worker2 # [ 5.142961] systemd[1]: Reached target Switch Root. worker2 # [ 5.144442] systemd[1]: Starting NixOS Activation... worker1 # [ 5.184591] initrd-nixos-activation-start[177]: booting system configuration /nix/store/r1caxc5yfxdfis0l5rpgjh307ha4z8wh-nixos-system-worker1-test worker1 # [ 5.216395] initrd-nixos-activation-start[177]: running activation script... lb # [ 5.198968] initrd-nixos-activation-start[180]: booting system configuration /nix/store/5hmvppg1fs8xkiccvl0m50n4dxjz92fr-nixos-system-lb-test client # [ 5.223326] initrd-nixos-activation-start[179]: booting system configuration /nix/store/jgg1bkcnnwpis3sdm7gl6kzqxlwc5ivn-nixos-system-client-test worker2 # [ 5.220742] initrd-nixos-activation-start[179]: booting system configuration /nix/store/azx0l63xg7ndaj4b8gzinjsgk55i1kf6-nixos-system-worker2-test lb # [ 5.229557] initrd-nixos-activation-start[180]: running activation script... client # [ 5.255556] initrd-nixos-activation-start[179]: running activation script... worker2 # [ 5.252222] initrd-nixos-activation-start[179]: running activation script... worker1 # [ 5.454121] initrd-nixos-activation-start[200]: setting up /etc... lb # [ 5.448390] initrd-nixos-activation-start[203]: setting up /etc... worker2 # [ 5.485961] initrd-nixos-activation-start[202]: setting up /etc... client # [ 5.495591] initrd-nixos-activation-start[202]: setting up /etc... worker1 # [ 5.568366] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker1 # [ 5.571324] systemd[1]: Finished NixOS Activation. worker1 # [ 5.572644] systemd[1]: Starting Switch Root... lb # [ 5.561925] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. lb # [ 5.564212] systemd[1]: Finished NixOS Activation. lb # [ 5.566107] systemd[1]: Starting Switch Root... worker1 # [ 5.595676] systemd[1]: Switching root. lb # [ 5.583534] systemd[1]: Switching root. worker2 # [ 5.597933] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker2 # [ 5.600834] systemd[1]: Finished NixOS Activation. worker2 # [ 5.601984] systemd[1]: Starting Switch Root... client # [ 5.607441] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. client # [ 5.610538] systemd[1]: Finished NixOS Activation. client # [ 5.611734] systemd[1]: Starting Switch Root... worker2 # [ 5.623552] systemd[1]: Switching root. client # [ 5.635928] systemd[1]: Switching root. worker1 # [ 5.784479] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). lb # [ 5.767186] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). client # [ 5.815018] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker2 # [ 5.811324] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker1 # [ 7.082659] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker1 # [ 7.089940] systemd[1]: Detected virtualization qemu. worker1 # [ 7.091860] systemd[1]: Detected architecture arm64. worker1 # [ 7.094062] systemd[1]: Detected first boot. worker1 # [ 7.097422] systemd[1]: Initializing machine ID from random generator. client # [ 7.070366] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 7.077411] systemd[1]: Detected virtualization qemu. client # [ 7.079235] systemd[1]: Detected architecture arm64. client # [ 7.081581] systemd[1]: Detected first boot. worker2 # [ 7.074971] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 7.084970] systemd[1]: Initializing machine ID from random generator. worker2 # [ 7.082161] systemd[1]: Detected virtualization qemu. worker2 # [ 7.084175] systemd[1]: Detected architecture arm64. worker2 # [ 7.086639] systemd[1]: Detected first boot. lb # [ 7.086451] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 7.090250] systemd[1]: Initializing machine ID from random generator. lb # [ 7.093746] systemd[1]: Detected virtualization qemu. lb # [ 7.095596] systemd[1]: Detected architecture arm64. lb # [ 7.097360] systemd[1]: Detected first boot. lb # [ 7.101221] systemd[1]: Initializing machine ID from random generator. worker1 # [ 7.375075] systemd[1]: bpf-restrict-fs: LSM BPF program attached client # [ 7.362878] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 7.362403] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 7.373397] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 7.527836] systemd[1]: Applying preset policy. worker2 # [ 7.510582] systemd[1]: Applying preset policy. client # [ 7.524436] systemd[1]: Applying preset policy. lb # [ 7.525693] systemd[1]: Applying preset policy. worker2 # [ 7.712420] systemd[1]: Populated /etc with preset unit settings. worker1 # [ 7.741640] systemd[1]: Populated /etc with preset unit settings. lb # [ 7.733923] systemd[1]: Populated /etc with preset unit settings. client # [ 7.738885] systemd[1]: Populated /etc with preset unit settings. worker1 # [ 7.942925] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker2 # [ 7.916063] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker1 # [ 7.944742] systemd[1]: Stopped initrd-switch-root.service. worker2 # [ 7.917728] systemd[1]: Stopped initrd-switch-root.service. worker1 # [ 7.947783] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker2 # [ 7.921375] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker1 # [ 7.950726] systemd[1]: Created slice Slice /system/getty. worker1 # [ 7.952832] systemd[1]: Created slice User and Session Slice. worker2 # [ 7.925662] systemd[1]: Created slice Slice /system/getty. worker1 # [ 7.954041] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker2 # [ 7.927764] systemd[1]: Created slice User and Session Slice. worker1 # [ 7.956418] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker2 # [ 7.929022] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 7.957751] systemd[1]: Expecting device /dev/hvc0... worker1 # [ 7.958910] systemd[1]: Expecting device /dev/ttyAMA0... worker2 # [ 7.930675] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker1 # [ 7.960184] systemd[1]: Reached target Local Encrypted Volumes. worker2 # [ 7.932595] systemd[1]: Expecting device /dev/hvc0... worker1 # [ 7.961426] systemd[1]: Stopped target initrd-fs.target. worker2 # [ 7.934147] systemd[1]: Expecting device /dev/ttyAMA0... client # [ 7.938782] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker1 # [ 7.963105] systemd[1]: Stopped target initrd-root-fs.target. worker2 # [ 7.935786] systemd[1]: Reached target Local Encrypted Volumes. worker1 # [ 7.964198] systemd[1]: Stopped target initrd-switch-root.target. lb # [ 7.939851] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker2 # [ 7.936910] systemd[1]: Stopped target initrd-fs.target. client # [ 7.940533] systemd[1]: Stopped initrd-switch-root.service. worker1 # [ 7.965525] systemd[1]: Reached target Virtual Machines and Containers. lb # [ 7.941543] systemd[1]: Stopped initrd-switch-root.service. worker2 # [ 7.938149] systemd[1]: Stopped target initrd-root-fs.target. worker1 # [ 7.967490] systemd[1]: Reached target Path Units. worker2 # [ 7.939524] systemd[1]: Stopped target initrd-switch-root.target. worker1 # [ 7.968463] systemd[1]: Reached target Remote File Systems. client # [ 7.944450] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker1 # [ 7.969746] systemd[1]: Reached target Slice Units. lb # [ 7.944759] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker2 # [ 7.941378] systemd[1]: Reached target Virtual Machines and Containers. worker2 # [ 7.943328] systemd[1]: Reached target Path Units. worker1 # [ 7.970984] systemd[1]: Reached target Swaps. worker2 # [ 7.944324] systemd[1]: Reached target Remote File Systems. client # [ 7.948900] systemd[1]: Created slice Slice /system/getty. lb # [ 7.948420] systemd[1]: Created slice Slice /system/getty. worker1 # [ 7.973991] systemd[1]: Listening on Query the User Interactively for a Password. worker2 # [ 7.945647] systemd[1]: Reached target Slice Units. client # [ 7.950965] systemd[1]: Created slice User and Session Slice. lb # [ 7.950586] systemd[1]: Created slice User and Session Slice. worker2 # [ 7.947555] systemd[1]: Reached target Swaps. lb # [ 7.951860] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 7.952248] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 7.977014] systemd[1]: Listening on Process Core Dump Socket. worker2 # [ 7.949896] systemd[1]: Listening on Query the User Interactively for a Password. lb # [ 7.953444] systemd[1]: Started Forward Password Requests to Wall Directory Watch. client # [ 7.954509] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker1 # [ 7.979159] systemd[1]: Listening on Credential Encryption/Decryption. lb # [ 7.955435] systemd[1]: Expecting device /dev/hvc0... client # [ 7.956517] systemd[1]: Expecting device /dev/hvc0... worker2 # [ 7.952888] systemd[1]: Listening on Process Core Dump Socket. worker1 # [ 7.981491] systemd[1]: Listening on Factory Reset Management. lb # [ 7.956973] systemd[1]: Expecting device /dev/ttyAMA0... client # [ 7.958205] systemd[1]: Expecting device /dev/ttyAMA0... worker1 # [ 7.982707] systemd[1]: Listening on Hostname Service Socket. worker2 # [ 7.955026] systemd[1]: Listening on Credential Encryption/Decryption. lb # [ 7.958685] systemd[1]: Reached target Local Encrypted Volumes. client # [ 7.959906] systemd[1]: Reached target Local Encrypted Volumes. worker2 # [ 7.957319] systemd[1]: Listening on Factory Reset Management. lb # [ 7.960333] systemd[1]: Stopped target initrd-fs.target. lb # [ 7.961946] systemd[1]: Stopped target initrd-root-fs.target. worker1 # [ 7.986748] systemd[1]: Starting Journal Log Access Socket... worker2 # [ 7.958593] systemd[1]: Listening on Hostname Service Socket. client # [ 7.961594] systemd[1]: Stopped target initrd-fs.target. client # [ 7.961910] systemd[1]: Stopped target initrd-root-fs.target. lb # [ 7.963066] systemd[1]: Stopped target initrd-switch-root.target. client # [ 7.962149] systemd[1]: Stopped target initrd-switch-root.target. worker1 # [ 7.988993] systemd[1]: Listening on Journal Audit Socket. lb # [ 7.964399] systemd[1]: Reached target Virtual Machines and Containers. client # [ 7.962395] systemd[1]: Reached target Virtual Machines and Containers. client # [ 7.962649] systemd[1]: Reached target Path Units. lb # [ 7.965886] systemd[1]: Reached target Path Units. client # [ 7.962893] systemd[1]: Reached target Remote File Systems. worker2 # [ 7.962888] systemd[1]: Starting Journal Log Access Socket... client # [ 7.963122] systemd[1]: Reached target Slice Units. lb # [ 7.966925] systemd[1]: Reached target Remote File Systems. worker1 # [ 7.991757] systemd[1]: Listening on Console Output Muting Service Socket. client # [ 7.963349] systemd[1]: Reached target Swaps. worker2 # [ 7.964560] systemd[1]: Listening on Journal Audit Socket. lb # [ 7.968138] systemd[1]: Reached target Slice Units. worker1 # [ 7.993268] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. lb # [ 7.969793] systemd[1]: Reached target Swaps. client # [ 7.970632] systemd[1]: Listening on Query the User Interactively for a Password. worker1 # [ 7.994675] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 7.967312] systemd[1]: Listening on Console Output Muting Service Socket. lb # [ 7.972178] systemd[1]: Listening on Query the User Interactively for a Password. client # [ 7.973633] systemd[1]: Listening on Process Core Dump Socket. worker1 # [ 7.997003] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 7.967931] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker2 # [ 7.968222] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os client # [ 7.975773] systemd[1]: Listening on Credential Encryption/Decryption. lb # [ 7.975062] systemd[1]: Listening on Process Core Dump Socket. worker2 # [ 7.968468] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki client # [ 7.978097] systemd[1]: Listening on Factory Reset Management. lb # [ 7.977154] systemd[1]: Listening on Credential Encryption/Decryption. worker1 # [ 8.002318] systemd[1]: Listening on Disk Repartitioning Service Socket. client # [ 7.979332] systemd[1]: Listening on Hostname Service Socket. worker1 # [ 8.003757] systemd[1]: Listening on udev Control Socket. lb # [ 7.979749] systemd[1]: Listening on Factory Reset Management. worker1 # [ 8.005219] systemd[1]: Listening on udev Varlink Socket. worker2 # [ 7.977329] systemd[1]: Listening on Disk Repartitioning Service Socket. lb # [ 7.980995] systemd[1]: Listening on Hostname Service Socket. client # [ 7.983088] systemd[1]: Starting Journal Log Access Socket... worker2 # [ 7.978785] systemd[1]: Listening on udev Control Socket. worker2 # [ 7.980156] systemd[1]: Listening on udev Varlink Socket. worker1 # [ 8.008622] systemd[1]: Mounting Huge Pages File System... lb # [ 7.984668] systemd[1]: Starting Journal Log Access Socket... client # [ 7.985173] systemd[1]: Listening on Journal Audit Socket. worker2 # [ 7.983505] systemd[1]: Mounting Huge Pages File System... lb # [ 7.986888] systemd[1]: Listening on Journal Audit Socket. client # [ 7.988602] systemd[1]: Listening on Console Output Muting Service Socket. client # [ 7.990233] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. lb # [ 7.989622] systemd[1]: Listening on Console Output Muting Service Socket. lb # [ 7.991248] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. client # [ 7.992425] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os lb # [ 7.992715] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 8.016865] systemd[1]: Mounting POSIX Message Queue File System... client # [ 7.994776] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki lb # [ 7.994552] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker1 # [ 8.020508] systemd[1]: Mounting Kernel Debug File System... worker2 # [ 7.991691] systemd[1]: Mounting POSIX Message Queue File System... client # [ 7.999902] systemd[1]: Listening on Disk Repartitioning Service Socket. worker2 # [ 7.996272] systemd[1]: Mounting Kernel Debug File System... client # [ 8.001271] systemd[1]: Listening on udev Control Socket. lb # [ 8.001298] systemd[1]: Listening on Disk Repartitioning Service Socket. client # [ 8.002657] systemd[1]: Listening on udev Varlink Socket. lb # [ 8.002851] systemd[1]: Listening on udev Control Socket. lb # [ 8.004202] systemd[1]: Listening on udev Varlink Socket. client # [ 8.005880] systemd[1]: Mounting Huge Pages File System... lb # [ 8.007423] systemd[1]: Mounting Huge Pages File System... worker1 # [ 8.033616] systemd[1]: Mounting Kernel Trace File System... worker2 # [ 8.007629] systemd[1]: Mounting Kernel Trace File System... client # [ 8.017649] systemd[1]: Mounting POSIX Message Queue File System... lb # [ 8.017830] systemd[1]: Mounting POSIX Message Queue File System... worker1 # [ 8.045303] systemd[1]: Starting Create List of Static Device Nodes... client # [ 8.023223] systemd[1]: Mounting Kernel Debug File System... lb # [ 8.022452] systemd[1]: Mounting Kernel Debug File System... worker1 # [ 8.045680] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 8.025315] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 8.026617] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 8.033269] systemd[1]: Mounting Kernel Trace File System... lb # [ 8.035050] systemd[1]: Mounting Kernel Trace File System... worker1 # [ 8.061648] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 8.062867] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm client # [ 8.043026] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 8.067849] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore client # [ 8.046181] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 8.042057] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 8.070679] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 8.046350] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 8.046874] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm lb # [ 8.053908] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 8.061091] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 8.056962] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore client # [ 8.063958] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm worker2 # [ 8.057328] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker1 # [ 8.088579] systemd[1]: Mounting FUSE Control File System... lb # [ 8.065347] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 8.088973] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 8.068801] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm client # [ 8.073752] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore client # [ 8.075670] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 8.077863] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore lb # [ 8.079767] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker2 # [ 8.078224] systemd[1]: Mounting FUSE Control File System... worker2 # [ 8.081376] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker1 # [ 8.117718] systemd[1]: Starting Journal Service... client # [ 8.097687] systemd[1]: Mounting FUSE Control File System... client # [ 8.098101] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 8.100858] systemd[1]: Mounting FUSE Control File System... lb # [ 8.101284] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker2 # [ 8.108022] systemd[1]: Starting Journal Service... client # [ 8.120512] systemd[1]: Starting Journal Service... worker2 # [ 8.118525] systemd[1]: Starting Load Kernel Modules... worker1 # [ 8.152108] systemd[1]: Starting Load Kernel Modules... lb # [ 8.131226] systemd[1]: Starting Journal Service... worker2 # [ 8.131685] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker1 # [ 8.165592] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker2 # [ 8.140231] systemd[1]: Starting Remount Root and Kernel File Systems... worker2 # [ 8.141938] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 8.170433] systemd[1]: Starting Remount Root and Kernel File Systems... lb # [ 8.145916] systemd[1]: Starting Load Kernel Modules... worker1 # [ 8.172668] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 8.177541] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 8.146923] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 8.149527] systemd[1]: Listening on Journal Log Access Socket. client # [ 8.154031] systemd[1]: Starting Load Kernel Modules... worker2 # [ 8.149951] systemd[1]: Mounted Huge Pages File System. worker1 # [ 8.181928] systemd[1]: Listening on Journal Log Access Socket. worker2 # [ 8.156871] systemd[1]: Mounted POSIX Message Queue File System. worker2 # [ 8.159071] systemd[1]: Mounted Kernel Debug File System. worker1 # [ 8.187373] systemd[1]: Mounted Huge Pages File System. worker2 # [ 8.161210] systemd[1]: Mounted Kernel Trace File System. client # [ 8.167281] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker1 # [ 8.189587] systemd[1]: Mounted POSIX Message Queue File System. worker1 # [ 8.190037] systemd[1]: Mounted Kernel Debug File System. worker1 # [ 8.190413] systemd[1]: Mounted Kernel Trace File System. worker1 # [ 8.190775] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 8.166659] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 8.167240] systemd[1]: Mounted FUSE Control File System. client # [ 8.174388] systemd-journald[272]: Collecting audit messages is enabled. lb # [ 8.173534] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker1 # [ 8.196546] systemd[1]: Mounted FUSE Control File System. client # [ 8.179980] systemd[1]: Starting Remount Root and Kernel File Systems... client # [ 8.184194] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 8.166309] systemd[1]: Queued start job for default target Multi-User System. worker2 # [ 8.181539] systemd[1]: Finished Create List of Static Device Nodes. client # [ 8.172527] systemd[1]: systemd-journald.service: Deactivated successfully. worker1 # [ 8.211733] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 8.192835] systemd[1]: Starting Remount Root and Kernel File Systems... worker1 # [ 8.220002] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 8.193208] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 8.202011] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 8.204918] systemd[1]: Starting Coldplug All udev Devices... lb # [ 8.220998] systemd[1]: Starting Coldplug All udev Devices... lb # [ 8.228484] systemd[1]: Listening on Journal Log Access Socket. client # [ 8.232346] systemd[1]: Started Journal Service. lb # [ 8.235323] systemd[1]: Mounted Huge Pages File System. lb # [ 8.240899] systemd[1]: Mounted POSIX Message Queue File System. lb # [ 8.243156] systemd[1]: Mounted Kernel Debug File System. client # [ 8.236404] systemd[1]: Listening on Journal Log Access Socket. client # [ 8.237507] systemd[1]: Mounted Huge Pages File System. lb # [ 8.254225] systemd[1]: Mounted Kernel Trace File System. client # [ 8.238300] systemd[1]: Mounted POSIX Message Queue File System. client # [ 8.239171] systemd[1]: Mounted Kernel Debug File System. lb # [ 8.256165] systemd-journald[273]: Collecting audit messages is enabled. worker1 # [ 8.285081] systemd[1]: Finished Load Kernel Modules. worker2 # [ 8.259158] systemd-journald[272]: Collecting audit messages is enabled. lb # [ 8.265631] systemd[1]: Finished Create List of Static Device Nodes. worker1 # [ 8.290211] systemd[1]: Starting Firewall... lb # [ 8.267246] systemd[1]: Mounted Kernel Configuration File System. client # [ 8.253622] systemd[1]: Mounted Kernel Trace File System. client # [ 8.254539] systemd[1]: Finished Create List of Static Device Nodes. client # [ 8.255447] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 8.296224] systemd[1]: Starting Apply Kernel Variables... lb # [ 8.273876] systemd[1]: Finished Load Kernel Modules. lb # [ 8.261468] systemd[1]: Queued start job for default target Multi-User System. lb # [ 8.265629] systemd[1]: systemd-journald.service: Deactivated successfully. lb # [ 8.266875] systemd-modules-load[274]: Module 'atkbd' is built in lb # [ 8.267882] systemd-modules-load[274]: Module 'loop' is built in worker1 # [ 8.310474] systemd-journald[270]: Collecting audit messages is enabled. lb # [ 8.287188] systemd[1]: Starting Firewall... client # [ 8.289664] EXT4-fs (vda): re-mounted 80cc710e-c6b4-4276-acd8-4081619fb465. worker2 # [ 8.285729] systemd[1]: Finished Load Kernel Modules. worker2 # [ 8.289434] EXT4-fs (vda): re-mounted 32fcf0a6-95eb-4478-af00-18bcbf63d3d3. worker2 # [ 8.294853] systemd[1]: Starting Firewall... lb # [ 8.299530] systemd[1]: Starting Apply Kernel Variables... client # [ 8.284642] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 8.288127] systemd[1]: Finished Remount Root and Kernel File Systems. client # [ 8.289129] systemd[1]: Listening on Disk Image Download Service Socket. worker2 # [ 8.303502] systemd[1]: Starting Apply Kernel Variables... lb # [ 8.308546] EXT4-fs (vda): re-mounted 80d3eab8-8922-4532-88a9-c338635f91f5. worker1 # [ 8.335124] EXT4-fs (vda): re-mounted d9b10c6c-1981-4594-a993-de61abb4c086. lb # [ 8.314343] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 8.300474] systemd-modules-load[274]: Module 'atkbd' is built in worker2 # [ 8.310798] systemd[1]: Finished Remount Root and Kernel File Systems. client # [ 8.301446] systemd-modules-load[274]: Module 'loop' is built in worker2 # [ 8.311479] systemd[1]: Listening on Disk Image Download Service Socket. client # [ 8.302338] systemd[1]: Starting Flush Journal to Persistent Storage... worker2 # [ 8.311794] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore client # [ 8.303285] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore lb # [ 8.324562] systemd[1]: Started Journal Service. worker1 # [ 8.349292] systemd[1]: Finished Remount Root and Kernel File Systems. worker1 # [ 8.349886] systemd[1]: Listening on Disk Image Download Service Socket. worker1 # [ 8.350177] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 8.327020] systemd[1]: Starting Load/Save OS Random Seed... worker2 # [ 8.329058] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 8.358931] systemd[1]: Starting Load/Save OS Random Seed... client # [ 8.319655] systemd[1]: Starting Load/Save OS Random Seed... worker2 # [ 8.333122] systemd[1]: Started Journal Service. worker1 # [ 8.360950] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 8.324329] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 8.321875] systemd[1]: Queued start job for default target Multi-User System. lb # [ 8.326055] systemd[1]: Mounted FUSE Control File System. worker2 # [ 8.323063] systemd[1]: systemd-journald.service: Deactivated successfully. worker1 # [ 8.370161] systemd[1]: Started Journal Service. lb # [ 8.328963] systemd-oomd[275]: No swap; memory pressure usage will be degraded lb # [ 8.336242] systemd[1]: Finished Remount Root and Kernel File Systems. client # [ 8.342797] systemd[1]: Finished Load Kernel Modules. worker1 # [ 8.369315] systemd[1]: Queued start job for default target Multi-User System. worker2 # [ 8.342186] systemd-modules-load[273]: Module 'atkbd' is built in client # [ 8.347255] systemd-oomd[275]: No swap; memory pressure usage will be degraded worker1 # [ 8.370537] systemd[1]: systemd-journald.service: Deactivated successfully. lb # [ 8.347779] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker2 # [ 8.347666] systemd-modules-load[273]: Module 'loop' is built in client # [ 8.354424] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 8.356168] systemd[1]: Starting Flush Journal to Persistent Storage... client # [ 8.363109] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker1 # [ 8.385645] systemd-modules-load[271]: Module 'atkbd' is built in worker1 # [ 8.392410] systemd-modules-load[271]: Module 'loop' is built in worker1 # [ 8.401204] systemd[1]: Starting Flush Journal to Persistent Storage... lb # [ 8.386546] systemd[1]: Listening on Disk Image Download Service Socket. client # [ 8.419646] systemd-journald[272]: Received client request to flush runtime journal. lb # [ 8.402298] systemd[1]: Starting Flush Journal to Persistent Storage... lb # [ 8.403350] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore lb # [ 8.417059] systemd[1]: Starting Load/Save OS Random Seed... lb # [ 8.417933] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 8.442106] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 8.423034] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 8.457215] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 8.431599] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 8.464644] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 8.449274] systemd-oomd[274]: No swap; memory pressure usage will be degraded worker2 # [ 8.476918] systemd-journald[272]: Received client request to flush runtime journal. client # [ 8.465537] systemd[1]: Mounted FUSE Control File System. client # [ 8.466542] systemd[1]: Finished Load/Save OS Random Seed. worker1 # [ 8.490768] systemd-oomd[273]: No swap; memory pressure usage will be degraded lb # [ 8.469701] systemd[1]: Finished Apply Kernel Variables. client # [ 8.472647] systemd[1]: Reached target First Boot Complete. client # [ 8.478088] systemd[1]: Finished Flush Journal to Persistent Storage. worker1 # [ 8.518799] systemd-journald[270]: Received client request to flush runtime journal. lb # [ 8.505764] systemd-journald[273]: Received client request to flush runtime journal. client # [ 8.509739] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 8.512294] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 8.518453] systemd[1]: Finished Load/Save OS Random Seed. worker2 # [ 8.519530] systemd[1]: Reached target First Boot Complete. worker2 # [ 8.530122] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker2 # [ 8.531155] systemd[1]: Finished Apply Kernel Variables. client # [ 8.536949] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 8.531943] systemd[1]: Finished Flush Journal to Persistent Storage. lb # [ 8.552723] systemd[1]: Finished Load/Save OS Random Seed. lb # [ 8.553836] systemd[1]: Reached target First Boot Complete. worker1 # [ 8.577250] systemd[1]: Finished Load/Save OS Random Seed. lb # [ 8.561501] systemd[1]: Finished Flush Journal to Persistent Storage. worker1 # [ 8.585553] systemd[1]: Reached target First Boot Complete. worker1 # [ 8.588938] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker1 # [ 8.591742] systemd[1]: Finished Flush Journal to Persistent Storage. lb # [ 8.579612] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 8.588904] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 8.625492] systemd[1]: Finished Create Static Device Nodes in /dev. worker1 # [ 8.628311] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 8.602085] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 8.603125] systemd[1]: Reached target Preparation for Local File Systems. worker1 # [ 8.632804] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker2 # [ 8.608288] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 8.708306] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 8.712062] systemd[1]: Reached target Preparation for Local File Systems. worker1 # [ 8.738385] systemd-udevd[309]: Using default interface naming scheme 'v261'. lb # [ 8.715754] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker2 # [ 8.725907] systemd-udevd[307]: Using default interface naming scheme 'v261'. worker2 # [ 8.834301] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 8.843178] systemd-udevd[316]: Using default interface naming scheme 'v261'. worker1 # [ 8.885510] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 8.928917] systemd[1]: Mounting /run/wrappers... worker2 # [ 8.905185] systemd[1]: Mounting /run/wrappers... lb # [ 8.929413] systemd[1]: Mounting /run/wrappers... worker1 # [ 8.974056] systemd[1]: Mounted /run/wrappers. worker1 # [ 8.974867] systemd[1]: Reached target Local File Systems. worker1 # [ 8.982674] systemd[1]: Listening on Boot Loader Control Service Socket. worker2 # [ 8.957506] systemd[1]: Mounted /run/wrappers. worker2 # [ 8.958307] systemd[1]: Reached target Local File Systems. worker2 # [ 8.961815] systemd[1]: Listening on Boot Loader Control Service Socket. worker1 # [ 8.992734] systemd[1]: Starting register-nix-paths.service... worker1 # [ 9.004486] systemd[1]: Starting Create SUID/SGID Wrappers... worker2 # [ 8.978418] systemd[1]: Starting register-nix-paths.service... lb # [ 8.983654] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 8.986275] systemd[1]: Mounted /run/wrappers. lb # [ 8.987002] systemd[1]: Reached target Local File Systems. worker2 # [ 8.983272] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 8.988748] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 8.990882] systemd[1]: Reached target Preparation for Local File Systems. worker1 # [ 9.014175] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker1 # [ 9.017356] systemd[1]: Starting Save Transient machine-id to Disk... worker1 # [ 9.019099] systemd[1]: Starting Create System Files and Directories... worker2 # [ 8.992592] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. lb # [ 8.997001] systemd[1]: Listening on Boot Loader Control Service Socket. client # [ 8.999566] systemd[1]: Mounting /run/wrappers... worker2 # [ 8.997484] systemd[1]: Starting Save Transient machine-id to Disk... lb # [ 9.001661] systemd[1]: Starting register-nix-paths.service... client # [ 9.004608] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker2 # [ 9.001958] systemd[1]: Starting Create System Files and Directories... lb # [ 9.012115] systemd[1]: Starting Create SUID/SGID Wrappers... lb # [ 9.013047] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. lb # [ 9.022078] systemd[1]: Starting Save Transient machine-id to Disk... lb # [ 9.026282] systemd[1]: Starting Create System Files and Directories... worker1 # [ 9.097616] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. client # [ 9.080781] systemd[1]: Mounted /run/wrappers. client # [ 9.081626] systemd[1]: Reached target Local File Systems. worker2 # [ 9.078619] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker1 # [ 9.107333] systemd[1]: Finished Save Transient machine-id to Disk. client # [ 9.091002] systemd[1]: Listening on Boot Loader Control Service Socket. worker2 # [ 9.088272] systemd[1]: Finished Save Transient machine-id to Disk. client # [ 9.096192] systemd[1]: Starting register-nix-paths.service... client # [ 9.099792] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 9.104299] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. client # [ 9.108395] systemd[1]: Starting Save Transient machine-id to Disk... client # [ 9.124124] systemd[1]: Starting Create System Files and Directories... lb # [ 9.168624] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker1 # [ 9.198186] systemd[1]: Finished Create System Files and Directories. client # [ 9.176677] systemd-udevd[303]: Using default interface naming scheme 'v261'. lb # [ 9.181228] systemd[1]: Finished Save Transient machine-id to Disk. worker1 # [ 9.208785] systemd[1]: Starting Rebuild Journal Catalog... worker1 # [ 9.217287] systemd[1]: Starting Record System Boot/Shutdown in UTMP... worker2 # [ 9.191941] systemd[1]: Finished Create System Files and Directories. worker2 # [ 9.199696] systemd[1]: Starting Rebuild Journal Catalog... worker2 # [ 9.216519] systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [ 9.226096] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. client # [ 9.237950] systemd[1]: Finished Save Transient machine-id to Disk. client # [ 9.242959] systemd[1]: Finished Coldplug All udev Devices. lb # [ 9.251652] systemd[1]: Finished Create System Files and Directories. lb # [ 9.266653] systemd[1]: Starting Rebuild Journal Catalog... lb # [ 9.280177] systemd[1]: Starting Record System Boot/Shutdown in UTMP... worker1 # [ 9.326758] systemd[1]: Finished Record System Boot/Shutdown in UTMP. worker2 # [ 9.299932] systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 9.312601] systemd[1]: Finished Create System Files and Directories. client # [ 9.324125] systemd[1]: Starting Rebuild Journal Catalog... client # [ 9.333059] systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [ 9.340186] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 9.388146] systemd[1]: Finished Rebuild Journal Catalog. worker1 # [ 9.393380] systemd[1]: Starting Update is Completed... worker2 # [ 9.370638] systemd[1]: Finished Rebuild Journal Catalog. worker2 # [ 9.374511] systemd[1]: Starting Update is Completed... lb # [ 9.401235] systemd[1]: Finished Record System Boot/Shutdown in UTMP. lb # [ 9.441765] systemd[1]: Finished Rebuild Journal Catalog. client # [ 9.448783] systemd[1]: Finished Record System Boot/Shutdown in UTMP. lb # [ 9.457726] systemd[1]: Starting Update is Completed... worker1 # [ 9.485471] systemd[1]: Finished Update is Completed. worker2 # [ 9.460247] systemd[1]: Finished Update is Completed. client # [ 9.468641] systemd[1]: Finished Rebuild Journal Catalog. client # [ 9.480438] systemd[1]: Starting Update is Completed... lb # [ 9.532442] systemd[1]: Finished Update is Completed. client # [ 9.544767] systemd[1]: Finished Update is Completed. worker1 # [ 9.809952] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker1 # [ 9.816455] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 9.810949] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 9.809049] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 9.812462] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 9.871186] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse client # [ 9.950236] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. client # [ 9.953563] systemd[1]: Finished Create SUID/SGID Wrappers. worker1 # [ 10.009228] systemd[1]: Finished Firewall. client # [ 10.031225] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker2 # [ 10.063282] systemd[1]: Finished Firewall. lb # [ 10.074786] systemd[1]: Finished Coldplug All udev Devices. client # [ 10.092136] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. lb # [ 10.154982] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 10.198719] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. lb # [ 10.202481] systemd[1]: Finished Create SUID/SGID Wrappers. lb # [ 10.209068] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse client # [ 10.271980] (udev-worker)[356]: Network interface NamePolicy= disabled on kernel command line. client # [ 10.296540] (udev-worker)[355]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 10.305244] (udev-worker)[355]: Network interface NamePolicy= disabled on kernel command line. lb # [ 10.316945] systemd[1]: Finished register-nix-paths.service. lb # [ 10.322889] systemd[1]: Reached target System Initialization. lb # [ 10.323880] systemd[1]: Started Discard unused filesystem blocks once a week. lb # [ 10.328508] systemd[1]: Started niks3 garbage collection timer. lb # [ 10.329493] systemd[1]: Started Daily Cleanup of Temporary Directories. lb # [ 10.330428] systemd[1]: Reached target Timer Units. lb # [ 10.331165] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 10.341259] systemd[1]: Listening on niks3 server socket. lb # [ 10.342102] systemd[1]: Listening on Nix Daemon Socket. lb # [ 10.342888] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. lb # [ 10.351953] systemd[1]: Reached target Socket Units. lb # [ 10.355675] systemd[1]: Reached target Basic System. worker2 # [ 10.353589] systemd[1]: Finished Coldplug All udev Devices. lb # [ 10.358760] systemd[1]: Starting Import lastlog data into lastlog2 database... lb # [ 10.359814] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 10.366814] systemd[1]: Starting Post-Boot Actions... worker1 # [ 10.393441] systemd[1]: Finished Coldplug All udev Devices. lb # [ 10.377307] systemd[1]: Started Reset console on configuration changes. lb # [ 10.408209] systemd[1]: Starting resolvconf update... client # [ 10.411504] systemd[1]: Finished register-nix-paths.service. client # [ 10.416442] systemd[1]: Reached target System Initialization. client # [ 10.417337] systemd[1]: Started Discard unused filesystem blocks once a week. client # [ 10.418424] systemd[1]: Started Daily Cleanup of Temporary Directories. client # [ 10.419380] systemd[1]: Reached target Timer Units. client # [ 10.427296] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 10.425602] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 10.431609] systemd[1]: Listening on Nix Daemon Socket. lb # [ 10.432143] systemd[1]: Started rustfs.service. client # [ 10.433083] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. client # [ 10.438437] systemd[1]: Reached target Socket Units. client # [ 10.440555] systemd[1]: Reached target Basic System. client # [ 10.445542] systemd[1]: Started backdoor.service. client # [ 10.446342] systemd[1]: Starting Import lastlog data into lastlog2 database... client # [ 10.450129] systemd[1]: Starting nftables firewall... worker1 # [ 10.475316] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 10.453067] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse client # [ 10.463688] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 10.465870] systemd[1]: Starting rustfs-bucket.service... client # [ 10.480723] systemd[1]: Starting Post-Boot Actions... worker1 # [ 10.508936] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker1 # [ 10.525301] systemd[1]: Finished register-nix-paths.service. lb # [ 10.500801] nsncd[465]: Sep 25 14:59:31.022 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 10.530120] systemd[1]: Reached target System Initialization. worker1 # [ 10.534119] systemd[1]: Started Discard unused filesystem blocks once a week. lb # [ 10.510892] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 10.519422] systemd[1]: Started Reset console on configuration changes. lb # [ 10.521911] systemd[1]: Finished Post-Boot Actions. worker1 # [ 10.554742] systemd[1]: Started Daily Cleanup of Temporary Directories. client # [ 10.534682] systemd[1]: Starting resolvconf update... worker2 # [ 10.540090] systemd[1]: Finished register-nix-paths.service. worker2 # [ 10.549509] systemd[1]: Reached target System Initialization. worker1 # [ 10.581541] systemd[1]: Reached target Timer Units. worker2 # [ 10.557371] systemd[1]: Started Discard unused filesystem blocks once a week. worker2 # [ 10.562950] systemd[1]: Started Daily Cleanup of Temporary Directories. lb # [ 10.568559] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker1 # [ 10.591938] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 10.572422] systemd[1]: Reached target Timer Units. worker1 # [ 10.601957] systemd[1]: Listening on Nix Daemon Socket. client # [ 10.579482] nsncd[397]: Sep 25 14:59:31.113 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 10.577063] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 10.610931] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. client # [ 10.589605] systemd[1]: Started Name Service Cache Daemon (nsncd). lb # [ 10.589010] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 10.588621] systemd[1]: Listening on Nix Daemon Socket. worker2 # [ 10.594726] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. lb # [ 10.599012] systemd[1]: Reached target Host and Network Name Lookups. client # connecting to host... lb # [ 10.607183] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 10.633492] systemd[1]: Reached target Socket Units. worker2 # [ 10.608614] systemd[1]: Reached target Socket Units. lb # [ 10.613066] systemd[1]: Started backdoor.service. client # [ 10.619357] systemd[1]: Finished Post-Boot Actions. worker2 # [ 10.616826] systemd[1]: Reached target Basic System. lb # [ 10.628295] systemd[1]: Starting D-Bus System Message Bus... worker1 # [ 10.652401] systemd[1]: Reached target Basic System. worker2 # [ 10.626604] systemd[1]: Starting Import lastlog data into lastlog2 database... client: Guest shell says: b'Spawning backdoor root shell...\n' worker2 # [ 10.638501] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 10.646465] systemd[1]: Starting Post-Boot Actions... worker1 # [ 10.680949] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 10.654485] systemd[1]: Starting User Login Management... lb # [ 10.662016] systemd[1]: Finished Import lastlog data into lastlog2 database. worker2 # [ 10.656148] systemd[1]: Started Reset console on configuration changes. client # [ 10.677113] systemd[1]: Condition check resulted in Virtio network device being skipped. worker2 # [ 10.664402] systemd[1]: Starting resolvconf update... worker1 # [ 10.696049] systemd[1]: Starting Import lastlog data into lastlog2 database... worker2 # [ 10.674704] systemd[1]: Starting D-Bus System Message Bus... worker1 # [ 10.713487] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 10.685791] systemd[1]: Finished Post-Boot Actions. worker1 # [ 10.734585] systemd[1]: Starting Post-Boot Actions... client # [ 10.678288] systemd[1]: Reached target Host and Network Name Lookups. client: connected to guest root shell worker2 # [ 10.709805] systemd[1]: Started Name Service Cache Daemon (nsncd). worker1 # [ 10.743335] systemd[1]: Started Reset console on configuration changes. client # [ 10.679130] systemd[1]: Reached target User and Group Name Lookups. client: (connecting took 11.14 seconds) client # [ 10.689049] systemd[1]: Starting D-Bus System Message Bus... client: (finished: waiting for the VM to finish booting, in 11.14 seconds) client # [ 10.705682] systemd[1]: Starting User Login Management... worker2 # [ 10.715476] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 10.723750] nsncd[488]: Sep 25 14:59:31.246 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 10.761203] systemd[1]: Starting resolvconf update... worker2 # [ 10.738268] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 10.742454] systemd[1]: Starting User Login Management... client # [ 10.754480] systemd[1]: Finished Import lastlog data into lastlog2 database. worker1 # [ 10.782101] nix-grpc-daemon[485]: ts=2026-09-25T14:59:31Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 10.760308] systemd[1]: Finished Import lastlog data into lastlog2 database. worker2 # [ 10.771528] dbus-broker-launch[492]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 10.777799] dbus-broker-launch[492]: NSS returned no entry for 'systemd-timesync' worker2 # [ 10.784204] dbus-broker-launch[492]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker1 # [ 10.819816] nix-grpc-daemon[485]: ts=2026-09-25T14:59:31Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 10.799398] systemd[1]: Started D-Bus System Message Bus. worker1 # [ 10.844399] nix-grpc-daemon[485]: ts=2026-09-25T14:59:31Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 10.849886] dbus-broker-launch[492]: Ready lb # connecting to host... worker1 # [ 10.892430] systemd[1]: Starting D-Bus System Message Bus... worker1 # [ 10.896918] nix-grpc-daemon[485]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 10.909190] nix-grpc-daemon[485]: E0000 00:00:1790348371.333709 485 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 10.887325] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker1 # [ 10.922381] nix-grpc-daemon[485]: E0000 00:00:1790348371.341793 485 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 10.934728] nix-grpc-daemon[485]: ts=2026-09-25T14:59:31Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 10.912379] systemd-logind[500]: New seat seat0. worker2 # [ 10.916951] systemd[1]: Started User Login Management. worker1 # [ 10.943116] nix-grpc-daemon[485]: ts=2026-09-25T14:59:31Z level=info event=scheduler_lock_lost error="error: http://lb:5751/api/farm/lead: Could not connect to server" worker2 # [ 10.923736] systemd[1]: Starting linger-users.service... worker1 # [ 10.955991] nix-grpc-daemon[485]: E0000 00:00:1790348371.358123 539 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. client # [ 10.939086] dbus-broker-launch[426]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 10.938415] systemd[1]: Stopped target Host and Network Name Lookups. worker1 # [ 10.968518] nix-grpc-daemon[485]: E0000 00:00:1790348371.387933 539 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 10.944595] systemd[1]: Stopping Host and Network Name Lookups... client # [ 10.954162] dbus-broker-launch[426]: NSS returned no entry for 'systemd-timesync' client # [ 10.955244] dbus-broker-launch[426]: Invalid user-name in /nix/store/6wvgzhp3xj1v1xybf1ixpbqi5gif5fbv-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker2 # [ 10.951596] systemd[1]: Stopped target User and Group Name Lookups. worker1 # [ 10.983995] nix-grpc-daemon[485]: E0000 00:00:1790348371.388735 539 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 10.960580] systemd[1]: Stopping User and Group Name Lookups... worker2 # [ 10.968273] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... client # [ 10.975651] systemd-logind[427]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 10.979540] systemd-logind[427]: New seat seat0. lb # [ 10.978709] systemd-logind[505]: New seat seat0. worker2 # [ 10.976487] systemd[1]: nscd.service: Deactivated successfully. worker1 # [ 10.994868] nix-grpc-daemon[485]: E0000 00:00:1790348371.414898 539 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. client # [ 10.988661] systemd[1]: Started User Login Management. worker2 # [ 10.983684] systemd[1]: Stopped Name Service Cache Daemon (nsncd). lb # [ 10.988894] systemd[1]: Started User Login Management. worker1 # [ 11.009969] nix-grpc-daemon[485]: ts=2026-09-25T14:59:31Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: FAILED_PRECONDITION: ipv4:192.168.1.2:50051: connect failed: addr: ipv4:192.168.1.2:50051 error: Network is unreachable" client # [ 11.009767] mousedev: PS/2 mouse device common for all mice client # [ 10.995246] systemd[1]: Starting linger-users.service... lb # [ 10.997167] systemd[1]: Starting linger-users.service... worker2 # [ 10.993369] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 10.996280] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. lb # [ 11.002127] dbus-broker-launch[503]: Looking up NSS user entry for 'systemd-timesync'... client # [ 11.004107] systemd[1]: Started D-Bus System Message Bus. worker1 # [ 11.027128] nsncd[487]: Sep 25 14:59:31.420 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 11.005364] systemd[1]: Started backdoor.service. lb # [ 11.010641] dbus-broker-launch[503]: NSS returned no entry for 'systemd-timesync' worker1 # [ 11.039661] systemd[1]: Finished Post-Boot Actions. lb # [ 11.020789] dbus-broker-launch[503]: Invalid user-name in /nix/store/4yiprls1sll47jid3xzq0m9nmi1qz2ww-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker1 # [ 11.046870] dbus-broker-launch[491]: Looking up NSS user entry for 'systemd-timesync'... worker1 # [ 11.052135] dbus-broker-launch[491]: NSS returned no entry for 'systemd-timesync' worker1 # [ 11.056795] dbus-broker-launch[491]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker2 # [ 11.038058] systemd[1]: linger-users.service: Deactivated successfully. worker1 # [ 11.068910] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 11.042877] systemd[1]: Finished linger-users.service. lb # [ 11.043972] systemd[1]: Stopped target Host and Network Name Lookups. lb # [ 11.049373] systemd[1]: Stopping Host and Network Name Lookups... worker1 # [ 11.072940] systemd[1]: Started Name Service Cache Daemon (nsncd). lb # [ 11.050242] systemd[1]: Stopped target User and Group Name Lookups. lb # [ 11.051133] systemd[1]: Stopping User and Group Name Lookups... lb # [ 11.051957] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker1 # [ 11.075742] systemd[1]: Reached target Host and Network Name Lookups. client # [ 11.056259] systemd[1]: Stopped target Host and Network Name Lookups. worker1 # [ 11.081772] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 11.083933] systemd[1]: Starting User Login Management... client # [ 11.061762] systemd[1]: Stopping Host and Network Name Lookups... client # [ 11.062759] systemd[1]: Stopped target User and Group Name Lookups. client # [ 11.063649] systemd[1]: Stopping User and Group Name Lookups... worker1 # [ 11.088304] systemd[1]: Finished Import lastlog data into lastlog2 database. lb # [ 11.071660] (udev-worker)[416]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 11.072335] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 11.077575] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... client # [ 11.078801] dbus-broker-launch[426]: Ready client # [ 11.079416] systemd[1]: nscd.service: Deactivated successfully. worker2 # [ 11.076454] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 11.093818] nix-grpc-daemon[485]: ts=2026-09-25T14:59:31Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: FAILED_PRECONDITION: ipv4:192.168.1.2:50051: connect failed: addr: ipv4:192.168.1.2:50051 error: Network is unreachable" worker1 # [ 11.107567] systemd[1]: Started D-Bus System Message Bus. worker1 # [ 11.109471] dbus-broker-launch[491]: Ready worker2 # [ 11.081634] nsncd[560]: Sep 25 14:59:31.616 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 11.090455] systemd[1]: Stopped Name Service Cache Daemon (nsncd). lb # [ 11.092790] (udev-worker)[416]: Network interface NamePolicy= disabled on kernel command line. lb # [ 11.093985] systemd[1]: nscd.service: Deactivated successfully. lb # [ 11.094791] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker2 # [ 11.091236] systemd[1]: Reached target User and Group Name Lookups. lb # [ 11.095948] (udev-worker)[339]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 11.122298] systemd-logind[558]: New seat seat0. worker2 # connecting to host... worker1 # [ 11.131011] systemd[1]: Stopped target Host and Network Name Lookups. worker2 # [ 11.105928] systemd[1]: Finished resolvconf update. worker2 # [ 11.106714] systemd[1]: Reached target Preparation for Network. client # [ 11.114565] systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 11.119280] systemd[1]: linger-users.service: Deactivated successfully. worker2 # [ 11.116190] systemd[1]: Starting DHCP Client... worker1 # [ 11.144304] systemd[1]: Stopping Host and Network Name Lookups... worker2 # [ 11.117008] systemd[1]: Starting Extra networking commands.... client # [ 11.122707] systemd[1]: Finished linger-users.service. lb # [ 11.125452] systemd[1]: Started D-Bus System Message Bus. lb # [ 11.130975] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 11.156241] systemd[1]: Stopped target User and Group Name Lookups. worker1 # [ 11.159492] systemd[1]: Stopping User and Group Name Lookups... worker1 # [ 11.163373] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker1 # [ 11.166286] systemd[1]: Started User Login Management. worker1 # [ 11.171080] systemd[1]: Starting linger-users.service... worker1 # [ 11.174377] systemd[1]: nscd.service: Deactivated successfully. lb # [ 11.155207] systemd[1]: linger-users.service: Deactivated successfully. worker1 # [ 11.179497] systemd[1]: Stopped Name Service Cache Daemon (nsncd). lb # [ 11.161147] dbus-broker-launch[503]: Ready worker1 # [ 11.184787] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 11.169264] systemd[1]: Finished linger-users.service. client # [ 11.198379] nsncd[494]: Sep 25 14:59:31.732 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 11.204307] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 11.205261] systemd[1]: Reached target Host and Network Name Lookups. client # [ 11.206170] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 11.230243] systemd[1]: linger-users.service: Deactivated successfully. worker1 # [ 11.233330] systemd[1]: Finished linger-users.service. worker1 # [ 11.241095] systemd[1]: Started Name Service Cache Daemon (nsncd). worker1 # [ 11.243816] nsncd[605]: Sep 25 14:59:31.772 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 11.223209] systemd[1]: Finished resolvconf update. lb # [ 11.223367] systemd[1]: Finished Firewall. worker1 # [ 11.253340] systemd[1]: Reached target Host and Network Name Lookups. client # [ 11.233058] systemd[1]: Starting DHCP Client... worker1 # [ 11.257032] systemd[1]: Reached target User and Group Name Lookups. lb # [ 11.237177] systemd[1]: Started Name Service Cache Daemon (nsncd). lb # [ 11.238490] nsncd[582]: Sep 25 14:59:31.759 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 11.264754] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker1 # [ 11.276922] systemd[1]: Finished resolvconf update. lb # [ 11.251313] systemd[1]: Reached target Host and Network Name Lookups. lb # [ 11.255465] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 11.281394] systemd[1]: Reached target Preparation for Network. lb # [ 11.263680] systemd[1]: Finished resolvconf update. worker1 # [ 11.288629] systemd[1]: Starting DHCP Client... worker1 # [ 11.292975] systemd[1]: Starting Extra networking commands.... lb # [ 11.269527] systemd[1]: Reached target Preparation for Network. lb # [ 11.279008] systemd[1]: Starting DHCP Client... lb # [ 11.287399] systemd[1]: Starting Extra networking commands.... worker1 # [ 11.324683] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 11.300503] (udev-worker)[336]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 11.330410] systemd[1]: Started backdoor.service. worker2 # [ 11.302834] (udev-worker)[336]: Network interface NamePolicy= disabled on kernel command line. client # [ 11.325925] systemd-logind[427]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) worker2 # [ 11.331598] (udev-worker)[341]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 11.345136] dhcpcd[595]: dhcpcd-10.3.2 starting worker2 # [ 11.356158] dhcpcd[638]: dev: loaded udev worker2 # [ 11.401217] 8021q: 802.1Q VLAN Support v1.8 client # [ 11.404857] dhcpcd[525]: dhcpcd-10.3.2 starting worker1 # connecting to host... client # [ 11.414997] dhcpcd[532]: dev: loaded udev worker2 # [ 11.423202] systemd[1]: Finished Extra networking commands.. worker2 # [ 11.425689] systemd[1]: Reached target Network. worker2 # [ 11.437769] systemd[1]: Starting Permit User Sessions... client # [ 11.463760] 8021q: 802.1Q VLAN Support v1.8 worker1 # [ 11.488401] nix-grpc-daemon[485]: ts=2026-09-25T14:59:32Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: FAILED_PRECONDITION: ipv4:192.168.1.2:50051: connect failed: addr: ipv4:192.168.1.2:50051 error: Network is unreachable" worker2 # [ 11.508348] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker1 # [ 11.545930] dhcpcd[640]: dhcpcd-10.3.2 starting worker2 # [ 11.519973] systemd[1]: Finished Permit User Sessions. worker2 # [ 11.524380] systemd[1]: Started Getty on tty1. worker2 # [ 11.525195] systemd[1]: Reached target Login Prompts. worker1 # [ 11.556003] dhcpcd[677]: dev: loaded udev worker2 # [ 11.556444] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker2 # [ 11.556963] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker2 # [ 11.559888] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker2 # [ 11.560212] cfg80211: failed to load regulatory.db client # [ 11.567052] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker1 # [ 11.605629] 8021q: 802.1Q VLAN Support v1.8 worker1 # [ 11.590134] (udev-worker)[343]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 11.596320] (udev-worker)[343]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 11.597851] (udev-worker)[477]: Network interface NamePolicy= disabled on kernel command line. lb # [ 11.597918] dhcpcd[628]: dhcpcd-10.3.2 starting client # [ 11.617877] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' client # [ 11.618385] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' client # [ 11.623871] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 client # [ 11.624190] cfg80211: failed to load regulatory.db worker2 # [ 11.622869] 8021q: adding VLAN 0 to HW filter on device eth0 lb # [ 11.610424] dhcpcd[660]: dev: loaded udev worker2 # [ 11.608337] dhcpcd[638]: eth0: waiting for carrier worker2 # [ 11.609152] dhcpcd[638]: eth0: carrier acquired lb # [ 11.621100] systemd[1]: Condition check resulted in Virtio network device being skipped. worker2 # [ 11.619301] dhcpcd[638]: DUID 00:01:00:01:32:49:48:d4:52:54:00:12:34:56 worker2 # [ 11.621802] dhcpcd[638]: eth0: IAID 00:12:34:56 client # [ 11.627167] systemd[1]: Finished nftables firewall. worker2 # [ 11.622492] dhcpcd[638]: eth0: adding address fe80::5054:ff:fe12:3456 client # [ 11.632431] systemd[1]: Reached target Preparation for Network. lb # [ 11.631924] systemd[1]: Starting Address configuration of eth1... client # [ 11.639138] systemd[1]: Starting Address configuration of eth1... client # [ 11.643955] systemd[1]: Starting Extra networking commands.... worker1 # [ 11.668859] systemd[1]: Finished Extra networking commands.. worker2 # [ 11.642721] systemd[1]: Condition check resulted in Virtio network device being skipped. worker2 # [ 11.648807] systemd[1]: Starting Address configuration of eth1... worker1 # [ 11.692546] systemd[1]: Reached target Network. worker1 # [ 11.693368] systemd[1]: Starting Permit User Sessions... worker1 # [ 11.721122] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker2 # [ 11.679499] dhcpcd[638]: eth0: IAID 00:12:34:56 lb # [ 11.722708] 8021q: 802.1Q VLAN Support v1.8 worker1 # [ 11.750877] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker1 # [ 11.752237] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker1 # [ 11.754065] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker1 # [ 11.754373] cfg80211: failed to load regulatory.db worker2 # [ 11.729927] 8021q: adding VLAN 0 to HW filter on device eth1 client # [ 11.737156] 8021q: adding VLAN 0 to HW filter on device eth0 client # [ 11.722769] dhcpcd[532]: eth0: waiting for carrier worker2 # [ 11.719988] systemd-logind[500]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 11.725602] dhcpcd[532]: libudev: received NULL device client # [ 11.727418] dhcpcd[532]: libudev: received NULL device worker2 # [ 11.726326] network-addresses-eth1-start[666]: adding address 192.168.1.4/24... done worker1 # [ 11.754365] systemd[1]: Finished Permit User Sessions. client # [ 11.732163] dhcpcd[532]: eth0: carrier acquired worker1 # [ 11.760647] systemd[1]: Started Getty on tty1. worker1 # [ 11.761364] systemd[1]: Reached target Login Prompts. worker2 # [ 11.737803] network-addresses-eth1-start[666]: adding address 2001:db8:1::4/64... done client # [ 11.748001] dhcpcd[532]: DUID 00:01:00:01:32:49:48:d4:52:54:00:12:34:56 client # [ 11.749032] dhcpcd[532]: eth0: IAID 00:12:34:56 client # [ 11.749726] dhcpcd[532]: eth0: adding address fe80::5054:ff:fe12:3456 lb # [ 11.751964] systemd-logind[505]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 11.774116] 8021q: adding VLAN 0 to HW filter on device eth1 worker2 # [ 11.752981] systemd[1]: Finished Address configuration of eth1. lb # [ 11.772384] systemd[1]: Finished Extra networking commands.. lb # [ 11.776991] systemd[1]: Reached target Network. client # [ 11.778651] network-addresses-eth1-start[548]: adding address 192.168.1.1/24... done worker1 # [ 11.829187] 8021q: adding VLAN 0 to HW filter on device eth0 worker1 # [ 11.814131] dhcpcd[677]: eth0: waiting for carrier lb # [ 11.792265] systemd[1]: Starting PostgreSQL Server... client # [ 11.795229] network-addresses-eth1-start[548]: adding address 2001:db8:1::1/64... done worker1 # [ 11.818344] dhcpcd[677]: eth0: waiting for carrier lb # [ 11.796289] systemd[1]: Starting Permit User Sessions... worker1 # [ 11.819339] dhcpcd[677]: eth0: carrier acquired worker1 # [ 11.823490] nix-grpc-daemon[485]: ts=2026-09-25T14:59:32Z level=info event=scheduler_lock_lost error="error: http://lb:5751/api/farm/lead: Could not connect to server" worker1 # [ 11.838579] systemd[1]: Condition check resulted in Virtio network device being skipped. worker1 # [ 11.839765] dhcpcd[677]: DUID 00:01:00:01:32:49:48:d4:52:54:00:12:34:56 worker1 # [ 11.845177] systemd[1]: Starting Address configuration of eth1... client # [ 11.825725] systemd[1]: Finished Address configuration of eth1. worker1 # [ 11.848570] dhcpcd[677]: eth0: IAID 00:12:34:56 worker1 # [ 11.849270] dhcpcd[677]: eth0: adding address fe80::5054:ff:fe12:3456 lb # [ 11.846121] 8021q: adding VLAN 0 to HW filter on device eth1 worker2 # [ 11.882515] mousedev: PS/2 mouse device common for all mice worker1 # [ 11.915280] 8021q: adding VLAN 0 to HW filter on device eth1 worker1 # [ 11.912563] network-addresses-eth1-start[716]: adding address 192.168.1.3/24... done lb # [ 11.892356] network-addresses-eth1-start[665]: adding address 192.168.1.2/24... done worker1 # [ 11.925549] network-addresses-eth1-start[716]: adding address 2001:db8:1::3/64... done client # [ 11.912873] systemd[1]: Finished Extra networking commands.. client # [ 11.917132] systemd[1]: Reached target Network. client # [ 11.920912] systemd[1]: Starting Permit User Sessions... worker1 # [ 11.943409] systemd[1]: Finished Address configuration of eth1. lb # [ 11.933552] network-addresses-eth1-start[665]: adding address 2001:db8:1::2/64... done lb # [ 11.956444] cfg80211: Loading compiled-in X.509 certificates for regulatory database lb # [ 11.962935] systemd[1]: Finished Permit User Sessions. client # [ 11.969521] systemd[1]: Finished Permit User Sessions. client # [ 11.976164] systemd[1]: Started Getty on tty1. client # [ 11.978825] systemd[1]: Reached target Login Prompts. lb # [ 11.980091] systemd[1]: Started Getty on tty1. lb # [ 11.980898] systemd[1]: Reached target Login Prompts. lb # [ 11.988085] systemd[1]: Finished Address configuration of eth1. worker2 # [ 11.984554] systemd-logind[500]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) lb # [ 12.027933] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' lb # [ 12.028442] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' lb # [ 12.032751] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 lb # [ 12.033072] cfg80211: failed to load regulatory.db worker2 # [ 12.046856] dhcpcd[638]: eth0: soliciting a DHCP lease worker1 # [ 12.077550] systemd-logind[558]: Watching system buttons on /dev/input/event0 (gpio-keys) worker2 # [ 12.052550] dhcpcd[638]: eth0: offered 10.0.2.15 from 10.0.2.2 client # [ 12.065961] dhcpcd[532]: eth0: soliciting a DHCP lease worker2 # [ 12.060930] dhcpcd[638]: eth0: probing address 10.0.2.15/24 client # [ 12.072606] dhcpcd[532]: eth0: offered 10.0.2.15 from 10.0.2.2 client # [ 12.080273] dhcpcd[532]: eth0: probing address 10.0.2.15/24 client: (finished: waiting for unit network-addresses-eth1.service, in 12.54 seconds) client: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; } lb # [ 12.166703] 8021q: adding VLAN 0 to HW filter on device eth0 lb # [ 12.151437] dhcpcd[660]: eth0: waiting for carrier lb # [ 12.156133] dhcpcd[660]: eth0: waiting for carrier lb # [ 12.160160] dhcpcd[660]: eth0: carrier acquired client: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; }, in 0.04 seconds) lb: waiting for unit network-addresses-eth1.service lb: waiting for the VM to finish booting lb: Guest shell says: b'Spawning backdoor root shell...\n' lb: connected to guest root shell lb: (connecting took 0.00 seconds) lb: (finished: waiting for the VM to finish booting, in 0.00 seconds) worker1 # [ 12.224874] mousedev: PS/2 mouse device common for all mice lb # [ 12.185778] dhcpcd[660]: DUID 00:01:00:01:32:49:48:d4:52:54:00:12:34:56 lb # [ 12.190690] dhcpcd[660]: eth0: IAID 00:12:34:56 lb # [ 12.191513] dhcpcd[660]: eth0: adding address fe80::5054:ff:fe12:3456 lb # [ 12.238481] mousedev: PS/2 mouse device common for all mice worker1 # [ 12.273878] systemd-logind[558]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) worker1 # [ 12.288803] nix-grpc-daemon[485]: ts=2026-09-25T14:59:32Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: FAILED_PRECONDITION: ipv6:%5B2001:db8:1::2%5D:50051: connect failed: addr: ipv6:%5B2001:db8:1::2%5D:50051 error: Network is unreachable" lb # [ 12.314916] postgresql-pre-start[716]: The files belonging to this database system will be owned by user "postgres". lb # [ 12.319636] postgresql-pre-start[716]: This user must also own the server process. lb: (finished: waiting for unit network-addresses-eth1.service, in 0.15 seconds) lb: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; } lb # [ 12.333016] postgresql-pre-start[716]: The database cluster will be initialized with locale "en_US.UTF-8". lb # [ 12.337844] postgresql-pre-start[716]: The default database encoding has accordingly been set to "UTF8". lb # [ 12.339119] postgresql-pre-start[716]: The default text search configuration will be set to "english". lb # [ 12.354089] postgresql-pre-start[716]: Data page checksums are enabled. lb # [ 12.355017] postgresql-pre-start[716]: fixing permissions on existing directory /var/lib/postgresql/18 ... ok lb # [ 12.363264] postgresql-pre-start[716]: creating subdirectories ... ok lb # [ 12.367054] postgresql-pre-start[716]: selecting dynamic shared memory implementation ... posix lb: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; }, in 0.11 seconds) worker1: waiting for unit network-addresses-eth1.service worker1: waiting for the VM to finish booting worker1: Guest shell says: b'Spawning backdoor root shell...\n' worker1: connected to guest root shell worker1: (connecting took 0.00 seconds) worker1: (finished: waiting for the VM to finish booting, in 0.00 seconds) client # [ 12.464103] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 worker1: (finished: waiting for unit network-addresses-eth1.service, in 0.07 seconds) worker1: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; } lb # [ 12.503498] postgresql-pre-start[716]: selecting default "max_connections" ... 100 worker1: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; }, in 0.04 seconds) worker2: waiting for unit network-addresses-eth1.service worker2: waiting for the VM to finish booting worker2: Guest shell says: b'Spawning backdoor root shell...\n' worker2: connected to guest root shell worker2: (connecting took 0.00 seconds) worker2: (finished: waiting for the VM to finish booting, in 0.00 seconds) lb # [ 12.547577] systemd-logind[505]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) worker2 # [ 12.620330] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 worker2: (finished: waiting for unit network-addresses-eth1.service, in 0.08 seconds) worker2: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; } lb # [ 12.637322] postgresql-pre-start[716]: selecting default "shared_buffers" ... 128MB worker2: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; }, in 0.04 seconds) lb: waiting for unit niks3.service lb # [ 12.740916] dhcpcd[660]: eth0: soliciting a DHCP lease lb # [ 12.744592] dhcpcd[660]: eth0: offered 10.0.2.15 from 10.0.2.2 lb # [ 12.752224] dhcpcd[660]: eth0: probing address 10.0.2.15/24 client # [ 12.782866] systemd[1]: Starting Virtual Console Setup... client # [ 12.825268] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 12.827591] systemd[1]: Stopped Virtual Console Setup. client # [ 12.836405] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. client # [ 12.844785] systemd[1]: Starting Virtual Console Setup... worker1 # [ 12.891380] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 client # [ 12.890626] systemd-logind[427]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) client # [ 12.967763] systemd-vconsole-setup[642]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 12.963694] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. client # [ 12.971921] systemd[1]: Finished Virtual Console Setup. worker2 # [ 12.974679] systemd[1]: Starting Virtual Console Setup... worker2 # [ 12.991004] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 12.998390] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 12.999176] systemd[1]: Starting Virtual Console Setup... worker2 # [ 13.074527] systemd-logind[500]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker2 # [ 13.123386] systemd-vconsole-setup[703]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 13.128309] systemd[1]: Finished Virtual Console Setup. worker1 # [ 13.184239] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker1 # [ 13.195889] systemd[1]: Starting Virtual Console Setup... worker1 # [ 13.214372] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 13.220216] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 13.222933] systemd[1]: Starting Virtual Console Setup... worker1 # [ 13.284427] systemd-logind[558]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker1 # [ 13.340625] systemd-vconsole-setup[755]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 13.344202] systemd[1]: Finished Virtual Console Setup. worker2 # [ 13.577322] dhcpcd[638]: eth0: soliciting an IPv6 router worker2 # [ 13.579435] dhcpcd[638]: eth0: Router Advertisement from fe80::2 worker2 # [ 13.581032] dhcpcd[638]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker2 # [ 13.582562] dhcpcd[638]: eth0: adding route to fec0::/64 worker2 # [ 13.583793] dhcpcd[638]: eth0: adding default route via fe80::2 lb # [ 13.612224] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 worker1 # [ 13.728561] dhcpcd[677]: eth0: soliciting a DHCP lease worker1 # [ 13.730462] dhcpcd[677]: eth0: offered 10.0.2.15 from 10.0.2.2 worker1 # [ 13.736497] dhcpcd[677]: eth0: probing address 10.0.2.15/24 client # [ 13.753934] dhcpcd[532]: eth0: soliciting an IPv6 router client # [ 13.755741] dhcpcd[532]: eth0: Router Advertisement from fe80::2 client # [ 13.757243] dhcpcd[532]: eth0: adding address fec0::5054:ff:fe12:3456/64 client # [ 13.758687] dhcpcd[532]: eth0: adding route to fec0::/64 client # [ 13.759888] dhcpcd[532]: eth0: adding default route via fe80::2 worker1 # [ 13.807031] dhcpcd[677]: eth0: soliciting an IPv6 router worker1 # [ 13.809244] dhcpcd[677]: eth0: Router Advertisement from fe80::2 worker1 # [ 13.810564] dhcpcd[677]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker1 # [ 13.811980] dhcpcd[677]: eth0: adding route to fec0::/64 worker1 # [ 13.813650] dhcpcd[677]: eth0: adding default route via fe80::2 worker1 # [ 13.889899] nix-grpc-daemon[485]: ts=2026-09-25T14:59:34Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2001:db8:1::2%5D:50051: Failed to connect to remote host: Connection refused" lb # [ 13.927315] dhcpcd[660]: eth0: soliciting an IPv6 router lb # [ 13.927792] dhcpcd[660]: eth0: Router Advertisement from fe80::2 lb # [ 13.927865] dhcpcd[660]: eth0: adding address fec0::5054:ff:fe12:3456/64 lb # [ 13.927928] dhcpcd[660]: eth0: adding route to fec0::/64 lb # [ 13.927977] dhcpcd[660]: eth0: adding default route via fe80::2 lb # [ 13.976293] systemd[1]: Starting Virtual Console Setup... lb # [ 14.009944] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. lb # [ 14.023522] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 14.028419] systemd[1]: Stopped Virtual Console Setup. lb # [ 14.041260] systemd[1]: Starting Virtual Console Setup... lb # [ 14.102453] systemd-logind[505]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) lb # [ 14.420352] postgresql-pre-start[716]: selecting default time zone ... UTC lb # [ 14.424169] postgresql-pre-start[716]: creating configuration files ... ok lb # [ 14.445659] systemd-vconsole-setup[769]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 14.450050] systemd[1]: Finished Virtual Console Setup. lb # [ 14.673481] postgresql-pre-start[716]: running bootstrap script ... ok lb # [ 15.311942] postgresql-pre-start[716]: performing post-bootstrap initialization ... ok lb # [ 15.505732] postgresql-pre-start[716]: syncing data to disk ... ok lb # [ 15.507781] postgresql-pre-start[716]: initdb: warning: enabling "trust" authentication for local connections lb # [ 15.509387] postgresql-pre-start[716]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. lb # [ 15.511452] postgresql-pre-start[716]: Success. You can now start the database server using: lb # [ 15.512947] postgresql-pre-start[716]: pg_ctl -D /var/lib/postgresql/18 -l logfile start lb # [ 15.607258] postgres[794]: [794] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit lb # [ 15.610530] postgres[794]: [794] LOG: listening on IPv6 address "::1", port 5432 lb # [ 15.611867] postgres[794]: [794] LOG: listening on IPv4 address "127.0.0.1", port 5432 lb # [ 15.614679] postgres[794]: [794] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432" lb # [ 15.628743] postgres[803]: [803] LOG: database system was shut down at 2026-09-25 14:59:35 GMT lb # [ 15.635401] postgres[794]: [794] LOG: database system is ready to accept connections lb # [ 15.639954] systemd[1]: Started PostgreSQL Server. lb # [ 15.650174] systemd[1]: Starting PostgreSQL Setup Scripts... lb # [ 15.813247] postgresql-setup-start[814]: CREATE DATABASE lb # [ 15.849976] postgresql-setup-start[819]: CREATE ROLE lb # [ 15.864624] postgresql-setup-start[821]: ALTER DATABASE lb # [ 15.870113] systemd[1]: Finished PostgreSQL Setup Scripts. lb # [ 15.872168] systemd[1]: Reached target PostgreSQL. client # [ 16.809630] dhcpcd[532]: eth0: leased 10.0.2.15 for 86400 seconds client # [ 16.809933] dhcpcd[532]: eth0: adding route to 10.0.2.0/24 client # [ 16.810010] dhcpcd[532]: eth0: adding default route via 10.0.2.2 worker2 # [ 16.859655] dhcpcd[638]: eth0: leased 10.0.2.15 for 86400 seconds worker2 # [ 16.859890] dhcpcd[638]: eth0: adding route to 10.0.2.0/24 worker2 # [ 16.859967] dhcpcd[638]: eth0: adding default route via 10.0.2.2 client # [ 16.918129] systemd[1]: Started DHCP Client. client # [ 16.920146] systemd[1]: Reached target Multi-User System. client # [ 16.923006] systemd[1]: Startup finished in 1.037s (kernel) + 5.611s (initrd) + 10.272s (userspace) = 16.921s. worker2 # [ 16.967247] systemd[1]: Started DHCP Client. worker2 # [ 16.970699] systemd[1]: Reached target Network is Online. worker2 # [ 16.974129] systemd[1]: Starting scheduler-token.service... worker2 # [ 17.017290] systemd[1]: scheduler-token.service: Main process exited, code=exited, status=7/NOTRUNNING worker2 # [ 17.020806] systemd[1]: scheduler-token.service: Failed with result 'exit-code'. worker2 # [ 17.022354] systemd[1]: Failed to start scheduler-token.service. worker2 # [ 17.023449] systemd[1]: Dependency failed for Nix worker-protocol over gRPC. worker2 # [ 17.025971] systemd[1]: nix-grpc-daemon.service: Job nix-grpc-daemon.service/start failed with result 'dependency'. worker2 # [ 17.027681] systemd[1]: scheduler-token.service: Consumed 27ms CPU time over 46ms wall clock time, 3.1M memory peak, 40B incoming IP traffic, 60B outgoing IP traffic. worker2 # [ 17.030185] systemd[1]: Reached target Multi-User System. worker2 # [ 17.030998] systemd[1]: Startup finished in 1.030s (kernel) + 5.622s (initrd) + 10.372s (userspace) = 17.025s. worker1 # [ 17.090407] nix-grpc-daemon[485]: ts=2026-09-25T14:59:37Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2001:db8:1::2%5D:50051: Failed to connect to remote host: Connection refused" lb # [ 17.670977] dhcpcd[660]: eth0: leased 10.0.2.15 for 86400 seconds lb # [ 17.671189] dhcpcd[660]: eth0: adding route to 10.0.2.0/24 lb # [ 17.671261] dhcpcd[660]: eth0: adding default route via 10.0.2.2 lb # [ 17.774448] systemd[1]: Started DHCP Client. lb # [ 17.776545] systemd[1]: Reached target Network is Online. lb # [ 17.780810] systemd[1]: Started Envoy reverse proxy. lb # [ 17.784761] systemd[1]: Started mock-oidc.service. lb # [ 17.892004] mock-oidc-server[898]: Mock OIDC Server running lb # [ 17.893327] mock-oidc-server[898]: OIDC Address: 192.168.1.2:8080 lb # [ 17.894590] mock-oidc-server[898]: Issue Address: 0.0.0.0:8081 lb # [ 17.895438] mock-oidc-server[898]: Issuer: http://192.168.1.2:8080/oidc lb # [ 17.896807] mock-oidc-server[898]: JWKS: http://192.168.1.2:8080/oidc/.well-known/jwks.json lb # [ 17.897920] mock-oidc-server[898]: Discovery: http://192.168.1.2:8080/oidc/.well-known/openid-configuration lb # [ 17.899394] mock-oidc-server[898]: Issue tokens: http://0.0.0.0:8081/issue?sub=... lb # [ 18.127113] envoy[897]: [2026-09-25 14:59:38.649][897][info][main] [source/server/server.cc:467] initializing epoch 0 (base id=0, hot restart version=11.120) lb # [ 18.130047] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:469] statically linked extensions: lb # [ 18.131560] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.clusters: envoy.cluster.dns, envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.composite, envoy.clusters.dynamic_forward_proxy, envoy.clusters.dynamic_modules, envoy.clusters.mcp_multicluster, envoy.clusters.redis, envoy.clusters.reverse_connection lb # [ 18.137188] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain lb # [ 18.139192] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.quic.client_packet_writer: envoy.quic.packet_writer.default lb # [ 18.141609] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.matching.action: envoy.matching.action.transport_socket.name, envoy.matching.actions.format_string, filter-chain-name lb # [ 18.144963] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.string_matcher: envoy.string_matcher.lua lb # [ 18.150902] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action lb # [ 18.153877] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher lb # [ 18.156895] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.regex_engines: envoy.regex_engines.google_re2 lb # [ 18.159032] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor worker2 # [ 18.161962] systemd[1]: scheduler-token.service: Scheduled restart job, restart counter is at 1. worker2 # [ 18.164245] systemd[1]: Starting scheduler-token.service... lb # [ 18.162661] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.network_namespace, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san lb # [ 18.187019] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.matching.transport_socket.input: envoy.matching.inputs.endpoint_metadata, envoy.matching.inputs.locality_metadata, envoy.matching.inputs.transport_socket_filter_state lb # [ 18.191360] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.tracers: envoy.tracers.datadog, envoy.tracers.dynamic_modules, envoy.tracers.fluentd, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin lb # [ 18.198583] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 18.207998] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.load_balancing_policies: envoy.load_balancing_policies.client_side_weighted_round_robin, envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.dynamic_modules, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.load_aware_locality, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.override_host, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset, envoy.load_balancing_policies.wrr_locality lb # [ 18.222546] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr lb # [ 18.225982] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] network.connection.client: default, envoy_internal lb # [ 18.228344] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case lb # [ 18.231038] envoy[897]: [2026-09-25 14:59:38.651][897][info][main] [source/server/server.cc:471] envoy.matching.generic_proxy_request_input.input: envoy.matching.generic_proxy.input.host, envoy.matching.generic_proxy.input.method, envoy.matching.generic_proxy.input.path, envoy.matching.generic_proxy.input.property, envoy.matching.generic_proxy.input.request, envoy.matching.generic_proxy.input.service lb # [ 18.236210] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.quic.connection_id_generator: envoy.quic.connection_id_generator.quic_lb, envoy.quic.deterministic_connection_id_generator lb # [ 18.239046] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment, envoy.tracers.opentelemetry.resource_detectors.static_config lb # [ 18.250569] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.quic.server_preferred_address: quic.server_preferred_address.datasource, quic.server_preferred_address.fixed worker2 # [ 18.251128] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 18.254695] systemd[1]: Finished scheduler-token.service. lb # [ 18.255215] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest worker2 # [ 18.255959] systemd[1]: scheduler-token.service: Consumed 34ms CPU time over 1.232s wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb # [ 18.263470] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter lb # [ 18.267093] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 18.272289] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.serializers: dubbo.hessian2 lb # [ 18.275114] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter lb # [ 18.279448] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator lb # [ 18.282670] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.protocols: dubbo lb # [ 18.285297] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme lb # [ 18.288833] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.dynamic_modules, envoy.filters.udp_listener.udp_proxy lb # [ 18.291529] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.retry_priorities: envoy.retry_priorities.previous_priorities lb # [ 18.294289] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple lb # [ 18.297158] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.http.ext_proc.response_processors: envoy.extensions.http.ext_proc.save_processing_response lb # [ 18.299422] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.grpc_credentials: envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata lb # [ 18.301817] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.geoip_providers: envoy.geoip_providers.maxmind lb # [ 18.303544] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.formatter: envoy.formatter.cel, envoy.formatter.dynamic_modules, envoy.formatter.file_content, envoy.formatter.generic_secret, envoy.formatter.metadata, envoy.formatter.req_without_query lb # [ 18.307033] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts lb # [ 18.310153] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua, envoy.router.cluster_specifier_plugin.matcher lb # [ 18.313045] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.tls.certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 18.315126] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 18.317587] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.http.header_validators: envoy.http.header_validators.envoy_default lb # [ 18.319554] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.ssl.server_context_factory: envoy.ssl.server_context_factory.default lb # [ 18.323806] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.filters.upstream_network: envoy.filters.upstream_network.reverse_tunnel_lifecycle lb # [ 18.326205] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.dynamic_metadata, envoy.matching.inputs.dynamic_module_data_input, envoy.matching.inputs.filter_state, envoy.matching.inputs.local_reply, envoy.matching.inputs.network_namespace, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params lb # [ 18.339906] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.fault, envoy.filters.http.a2a, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.ai_protocol_manager, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.api_key_auth, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.bandwidth_share, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cache_v2, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_server, envoy.filters.http.file_system_buffer, envoy.filters.http.filter_chain, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_reverse_transcoder, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.mcp, envoy.filters.http.mcp_json_rest_bridge, envoy.filters.http.mcp_router, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.proto_api_scrubber, envoy.filters.http.proto_message_extraction, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.sse_to_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.thrift_to_metadata, envoy.filters.http.transform, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router lb # [ 18.371939] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff lb # [ 18.378149] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.upstreams: envoy.filters.connection_pools.tcp.generic lb # [ 18.380004] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router lb # [ 18.383255] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.common.key_value: envoy.key_value.file_based lb # [ 18.386151] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port lb # [ 18.387966] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.resolvers: envoy.ip, envoy.resolvers.reverse_connection lb # [ 18.390025] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation lb # [ 18.392134] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.generic_proxy.filters: envoy.filters.generic.router lb # [ 18.393968] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.health_check.event_sinks: envoy.health_check.event_sink.file lb # [ 18.395842] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable lb # [ 18.398122] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.credential_injector, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec, envoy.filters.http.upstream_rbac, envoy.filters.http.wasm lb # [ 18.410848] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.dynamic_modules, envoy.tls.cert_validator.spiffe lb # [ 18.416627] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.filters.listener: envoy.filters.listener.dynamic_modules, envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.set_filter_state, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector lb # [ 18.423905] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.http.cache_v2: envoy.extensions.http.cache_v2.file_system_http_cache, envoy.extensions.http.cache_v2.simple lb # [ 18.426656] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy lb # [ 18.430244] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo, envoy.network.dns_resolver.hickory lb # [ 18.433164] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.rbac.principals: envoy.rbac.principals.mtls_authenticated lb # [ 18.435001] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel, envoy.access_loggers.extension_filters.process_ratelimit lb # [ 18.437994] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.health_checkers: envoy.health_checkers.dynamic_modules, envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift lb # [ 18.441531] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.tls.certificate_mappers: envoy.tls.certificate_mappers.sni, envoy.tls.certificate_mappers.static_name lb # [ 18.443844] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.rds_factory: envoy.rds_factory.default lb # [ 18.445776] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector, envoy.upstream.local_address_selector.filter_state_override lb # [ 18.449045] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router lb # [ 18.450792] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation lb # [ 18.453416] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.upstream_options: envoy.extensions.upstreams.http.reverse_tunnel.v3.ReverseTunnelUpstreamCodecOptions, envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options lb # [ 18.459076] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche lb # [ 18.461183] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso lb # [ 18.463267] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls lb # [ 18.469984] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.route_config_update_requester: envoy.route_config_update_requester.default lb # [ 18.472158] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.content_parsers: envoy.content_parsers.json lb # [ 18.473905] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.route.early_data_policy: envoy.route.early_data_policy.default lb # [ 18.475962] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory lb # [ 18.480024] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.ext_authz, envoy.filters.udp.session.http_capsule lb # [ 18.482867] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.bootstrap: envoy.bootstrap.dynamic_modules, envoy.bootstrap.internal_listener, envoy.bootstrap.reverse_tunnel.downstream_socket_interface, envoy.bootstrap.reverse_tunnel.upstream_socket_interface, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface, envoy.extensions.network.socket_interface.sockmap lb # [ 18.489096] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.matching.stat_matching_data.input: stat_full_name_match_input lb # [ 18.491035] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.srds_factory: envoy.srds_factory.default lb # [ 18.492867] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] quic.http_server_connection: quic.http_server_connection.default lb # [ 18.494731] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.access_loggers: envoy.access_loggers.dynamic_modules, envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stats, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log lb # [ 18.502456] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_mappers: envoy.tls.certificate_mappers.static_name, envoy.tls.upstream_certificate_mappers.filter_state_override lb # [ 18.506315] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.thrift_proxy.transports: auto, framed, header, unframed lb # [ 18.508226] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.cel, envoy.tracers.opentelemetry.samplers.dynatrace, envoy.tracers.opentelemetry.samplers.parent_based, envoy.tracers.opentelemetry.samplers.trace_id_ratio_based lb # [ 18.512600] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.dynamic_modules, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.ext_proc, envoy.filters.network.generic_proxy, envoy.filters.network.geoip, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.reverse_tunnel, envoy.filters.network.reverse_tunnel_drain_aware_http_connection_manager, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_bandwidth_limit, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy lb # [ 18.526672] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.envelope, envoy.http.stateful_session.header lb # [ 18.529838] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.matching.stat_tag_matching_data.input: stat_tag_value_input lb # [ 18.531744] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls lb # [ 18.536997] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8 lb # [ 18.538882] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.http.forward_client_cert_details: forward_client_cert lb # [ 18.540809] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.matching.input_matchers: envoy.matching.input_matchers.generic_request_matcher, envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.dynamic_modules, envoy.matching.matchers.ip, envoy.matching.matchers.metadata_matcher, envoy.matching.matchers.runtime_fraction lb # [ 18.545471] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.http.injected_credentials: envoy.http.injected_credentials.generic, envoy.http.injected_credentials.oauth2 lb # [ 18.547786] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.quic.connection_debug_visitor: envoy.quic.connection_debug_visitor.basic, envoy.quic.connection_debug_visitor.quic_stats lb # [ 18.550513] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.hashable_string, envoy.network.application_protocols, envoy.network.ip, envoy.network.network_namespace, envoy.network.transport_socket.http_11_proxy.info, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.ratelimit.hits_addend, envoy.router.debug_config, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port lb # [ 18.559098] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.built_in_formatters: envoy.built_in_formatters.cel, envoy.built_in_formatters.http.default, envoy.built_in_formatters.metadata, envoy.built_in_formatters.stream_info.default, envoy.built_in_formatters.xfcc_value lb # [ 18.562902] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.request_id: envoy.request_id.uuid lb # [ 18.564599] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.generic_proxy.codecs: envoy.generic_proxy.codecs.dubbo, envoy.generic_proxy.codecs.http1 lb # [ 18.566782] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.connection_handler: envoy.connection_handler.default lb # [ 18.568658] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.http.ext_proc.processing_request_modifiers: envoy.extensions.http.ext_proc.mapped_attribute_builder lb # [ 18.571006] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.dynamic_modules, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd lb # [ 18.577014] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor lb # [ 18.579718] envoy[897]: [2026-09-25 14:59:38.652][897][info][main] [source/server/server.cc:471] envoy.resource_monitors: envoy.resource_monitors.cgroup_memory, envoy.resource_monitors.cpu_utilization, envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource lb # [ 18.583191] envoy[897]: [2026-09-25 14:59:38.673][897][info][main] [source/server/server.cc:531] HTTP header map info: lb # [ 18.585161] envoy[897]: [2026-09-25 14:59:38.673][897][info][main] [source/server/server.cc:534] request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-host,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-request-id lb # [ 18.601154] envoy[897]: [2026-09-25 14:59:38.673][897][info][main] [source/server/server.cc:534] request trailer map: 120 bytes: lb # [ 18.602684] envoy[897]: [2026-09-25 14:59:38.673][897][info][main] [source/server/server.cc:534] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id lb # [ 18.614773] envoy[897]: [2026-09-25 14:59:38.673][897][info][main] [source/server/server.cc:534] response trailer map: 144 bytes: grpc-message,grpc-status lb # [ 18.619888] envoy[897]: [2026-09-25 14:59:38.708][897][info][main] [source/server/server.cc:968] runtime: {} lb # [ 18.621369] envoy[897]: [2026-09-25 14:59:38.713][897][info][main] [source/server/server.cc:420] Raised file descriptor limits from 1024 to 524288. lb # [ 18.623057] envoy[897]: [2026-09-25 14:59:38.713][897][info][main] [source/server/server.cc:796] Starting admin HTTP server at 127.0.0.1:9901 lb # [ 18.624841] envoy[897]: [2026-09-25 14:59:38.713][897][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:9901 lb # [ 18.626354] envoy[897]: [2026-09-25 14:59:38.714][897][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration lb # [ 18.627955] envoy[897]: [2026-09-25 14:59:38.714][897][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s) lb # [ 18.629726] envoy[897]: [2026-09-25 14:59:38.714][897][info][config] [source/server/configuration_impl.cc:138] loading 2 cluster(s) lb # [ 18.631314] envoy[897]: [2026-09-25 14:59:38.784][897][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s) lb # [ 18.633396] envoy[897]: [2026-09-25 14:59:38.793][897][info][config] [source/server/configuration_impl.cc:164] loading stats configuration lb # [ 18.634980] envoy[897]: [2026-09-25 14:59:38.794][897][warning][main] [source/server/server.cc:1038] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor. lb # [ 18.638332] envoy[897]: [2026-09-25 14:59:38.794][897][info][main] [source/server/server.cc:1077] starting main dispatch loop lb # [ 18.639782] envoy[897]: [2026-09-25 14:59:38.794][897][info][runtime] [source/common/runtime/runtime_impl.cc:567] RTDS has finished initialization lb # [ 18.641613] envoy[897]: [2026-09-25 14:59:38.794][897][info][upstream] [source/common/upstream/cluster_manager_impl.cc:227] cm init: initializing secondary clusters worker1 # [ 18.756242] dhcpcd[677]: eth0: leased 10.0.2.15 for 86400 seconds worker1 # [ 18.757983] dhcpcd[677]: eth0: adding route to 10.0.2.0/24 worker1 # [ 18.759221] dhcpcd[677]: eth0: adding default route via 10.0.2.2 worker1 # [ 18.862208] systemd[1]: Started DHCP Client. worker1 # [ 18.863696] systemd[1]: Reached target Multi-User System. worker1 # [ 18.865073] systemd[1]: Startup finished in 1.037s (kernel) + 5.621s (initrd) + 12.204s (userspace) = 18.863s. worker1 # [ 22.090903] nix-grpc-daemon[485]: ts=2026-09-25T14:59:42Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused" worker1 # [ 27.091362] nix-grpc-daemon[485]: ts=2026-09-25T14:59:47Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused" lb # [ 31.227722] rustfs-bucket-start[1024]: mb s3://farm lb # [ 31.239656] systemd[1]: Finished rustfs-bucket.service. lb # [ 31.249225] systemd[1]: Starting niks3 server... lb # [ 31.416258] postgres[1044]: [1044] ERROR: relation "goose_db_version" does not exist at character 36 lb # [ 31.418329] postgres[1044]: [1044] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC lb # [ 31.450763] niks3-server[1035]: 2026/09/25 14:59:51 OK 20241026095416_initial_model.sql (22.29ms) lb # [ 31.453881] niks3-server[1035]: 2026/09/25 14:59:51 OK 20251210153512_drop_unused_gin_index.sql (3.42ms) lb # [ 31.459138] niks3-server[1035]: 2026/09/25 14:59:51 OK 20251218171726_add_pins.sql (5.19ms) lb # [ 31.463590] niks3-server[1035]: 2026/09/25 14:59:51 OK 20260628120000_add_object_size_and_stats.sql (4.42ms) lb # [ 31.468265] niks3-server[1035]: 2026/09/25 14:59:51 OK 20260905000000_add_claims.sql (4.63ms) lb # [ 31.471739] niks3-server[1035]: 2026/09/25 14:59:51 OK 20260920000000_drop_claims.sql (3.44ms) lb # [ 31.474416] niks3-server[1035]: 2026/09/25 14:59:51 OK 20260923120000_add_pushes.sql (2.59ms) lb # [ 31.476737] niks3-server[1035]: 2026/09/25 14:59:51 goose: successfully migrated database to version: 20260923120000 lb # [ 31.479583] niks3-server[1035]: 2026/09/25 14:59:52 OK 1_commit_pending_closure.sql (5.07ms) lb # [ 31.482058] niks3-server[1035]: 2026/09/25 14:59:52 OK 2_object_stats_trigger.sql (2.48ms) lb # [ 31.484660] niks3-server[1035]: 2026/09/25 14:59:52 OK 3_commit_push.sql (2.17ms) lb # [ 31.486132] niks3-server[1035]: 2026/09/25 14:59:52 goose: up to current file version: 3 lb # [ 31.487753] niks3-server[1035]: 2026/09/25 14:59:52 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key lb # [ 31.510142] niks3-server[1035]: 2026/09/25 14:59:52 INFO Read proxy enabled — serving cache objects from S3 lb # [ 31.511567] niks3-server[1035]: 2026/09/25 14:59:52 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 31.519704] niks3-server[1035]: 2026/09/25 14:59:52 INFO systemd watchdog enabled interval=15s lb # [ 31.521897] niks3-server[1035]: 2026/09/25 14:59:52 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 31.523419] systemd[1]: Started niks3 server. lb # [ 31.524136] systemd[1]: Reached target Multi-User System. lb # [ 31.524938] systemd[1]: Startup finished in 1.050s (kernel) + 5.603s (initrd) + 24.863s (userspace) = 31.517s. worker1 # [ 32.091879] nix-grpc-daemon[485]: ts=2026-09-25T14:59:52Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused" lb: (finished: waiting for unit niks3.service, in 19.85 seconds) lb: waiting for TCP port 5751 on localhost lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded! lb: (finished: waiting for TCP port 5751 on localhost, in 0.04 seconds) lb: waiting for unit envoy.service lb: (finished: waiting for unit envoy.service, in 0.03 seconds) lb: waiting for TCP port 50051 on localhost lb # [ 33.272945] envoy[897]: [2026-09-25 14:59:53.794][897][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:130] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment lb # [ 33.278231] envoy[897]: [2026-09-25 14:59:53.795][897][info][upstream] [source/common/upstream/cluster_manager_impl.cc:250] cm init: all clusters initialized lb # [ 33.278309] envoy[897]: [2026-09-25 14:59:53.795][897][info][main] [source/server/server.cc:1057] all clusters initialized. initializing init manager lb # [ 33.280706] envoy[897]: [2026-09-25 14:59:53.803][897][info][config] [source/common/listener_manager/listener_manager_impl.cc:1067] all dependencies initialized. starting workers lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded! lb # [ 33.631485] envoy[897]: conn peer=[::1]:36900 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=4 lb: (finished: waiting for TCP port 50051 on localhost, in 1.06 seconds) worker2 # [ 33.716782] systemd[1]: Starting scheduler-token.service... worker2 # [ 33.767280] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 33.772285] systemd[1]: Finished scheduler-token.service. worker2 # [ 33.773595] systemd[1]: scheduler-token.service: Consumed 32ms CPU time over 53ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 33.787718] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 33.899488] nix-grpc-daemon[786]: ts=2026-09-25T14:59:54Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 33.908459] nix-grpc-daemon[786]: ts=2026-09-25T14:59:54Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 33.910175] nix-grpc-daemon[786]: ts=2026-09-25T14:59:54Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 33.923017] nix-grpc-daemon[786]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 33.926071] nix-grpc-daemon[786]: E0000 00:00:1790348394.465165 786 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 33.930899] nix-grpc-daemon[786]: E0000 00:00:1790348394.473304 786 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 33.936094] nix-grpc-daemon[786]: ts=2026-09-25T14:59:54Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 33.945394] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 33.949670] nix-grpc-daemon[786]: E0000 00:00:1790348394.491067 828 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 33.976122] nix-grpc-daemon[786]: E0000 00:00:1790348394.509927 828 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 33.978527] nix-grpc-daemon[786]: E0000 00:00:1790348394.510313 828 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 33.989462] nix-grpc-daemon[786]: E0000 00:00:1790348394.530382 828 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2 # [ 34.003791] nix-grpc-daemon[786]: ts=2026-09-25T14:59:54Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.005770] nix-grpc-daemon[786]: ts=2026-09-25T14:59:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.206504] nix-grpc-daemon[786]: ts=2026-09-25T14:59:54Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.209321] nix-grpc-daemon[786]: ts=2026-09-25T14:59:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.410026] nix-grpc-daemon[786]: ts=2026-09-25T14:59:54Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.412886] nix-grpc-daemon[786]: ts=2026-09-25T14:59:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.613676] nix-grpc-daemon[786]: ts=2026-09-25T14:59:55Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.616395] nix-grpc-daemon[786]: ts=2026-09-25T14:59:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.817230] nix-grpc-daemon[786]: ts=2026-09-25T14:59:55Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.820237] nix-grpc-daemon[786]: ts=2026-09-25T14:59:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 35.021323] nix-grpc-daemon[786]: ts=2026-09-25T14:59:55Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.024068] nix-grpc-daemon[786]: ts=2026-09-25T14:59:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2 # [ 35.224775] nix-grpc-daemon[786]: ts=2026-09-25T14:59:55Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.227472] nix-grpc-daemon[786]: ts=2026-09-25T14:59:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.428295] nix-grpc-daemon[786]: ts=2026-09-25T14:59:55Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.431017] nix-grpc-daemon[786]: ts=2026-09-25T14:59:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.631662] nix-grpc-daemon[786]: ts=2026-09-25T14:59:56Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.634502] nix-grpc-daemon[786]: ts=2026-09-25T14:59:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.835056] nix-grpc-daemon[786]: ts=2026-09-25T14:59:56Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.837829] nix-grpc-daemon[786]: ts=2026-09-25T14:59:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 36.039488] nix-grpc-daemon[786]: ts=2026-09-25T14:59:56Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.041796] nix-grpc-daemon[786]: ts=2026-09-25T14:59:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2 # [ 36.239962] nix-grpc-daemon[786]: ts=2026-09-25T14:59:56Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.242877] nix-grpc-daemon[786]: ts=2026-09-25T14:59:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.443507] nix-grpc-daemon[786]: ts=2026-09-25T14:59:56Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.446476] nix-grpc-daemon[786]: ts=2026-09-25T14:59:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.647198] nix-grpc-daemon[786]: ts=2026-09-25T14:59:57Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.650184] nix-grpc-daemon[786]: ts=2026-09-25T14:59:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.850890] nix-grpc-daemon[786]: ts=2026-09-25T14:59:57Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.853845] nix-grpc-daemon[786]: ts=2026-09-25T14:59:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.055434] nix-grpc-daemon[786]: ts=2026-09-25T14:59:57Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.057761] nix-grpc-daemon[786]: ts=2026-09-25T14:59:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 37.092549] nix-grpc-daemon[485]: ts=2026-09-25T14:59:57Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2 # [ 37.256185] nix-grpc-daemon[786]: ts=2026-09-25T14:59:57Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.258976] nix-grpc-daemon[786]: ts=2026-09-25T14:59:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.459598] nix-grpc-daemon[786]: ts=2026-09-25T14:59:58Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.462465] nix-grpc-daemon[786]: ts=2026-09-25T14:59:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.663164] nix-grpc-daemon[786]: ts=2026-09-25T14:59:58Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.666152] nix-grpc-daemon[786]: ts=2026-09-25T14:59:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.866909] nix-grpc-daemon[786]: ts=2026-09-25T14:59:58Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.869718] nix-grpc-daemon[786]: ts=2026-09-25T14:59:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.071555] nix-grpc-daemon[786]: ts=2026-09-25T14:59:58Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.073933] nix-grpc-daemon[786]: ts=2026-09-25T14:59:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2 # [ 38.272280] nix-grpc-daemon[786]: ts=2026-09-25T14:59:58Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.275174] nix-grpc-daemon[786]: ts=2026-09-25T14:59:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.475940] nix-grpc-daemon[786]: ts=2026-09-25T14:59:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.478802] nix-grpc-daemon[786]: ts=2026-09-25T14:59:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.679412] nix-grpc-daemon[786]: ts=2026-09-25T14:59:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.682400] nix-grpc-daemon[786]: ts=2026-09-25T14:59:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.883066] nix-grpc-daemon[786]: ts=2026-09-25T14:59:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.885911] nix-grpc-daemon[786]: ts=2026-09-25T14:59:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 39.087642] nix-grpc-daemon[786]: ts=2026-09-25T14:59:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.090008] nix-grpc-daemon[786]: ts=2026-09-25T14:59:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2 # [ 39.288421] nix-grpc-daemon[786]: ts=2026-09-25T14:59:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.291242] nix-grpc-daemon[786]: ts=2026-09-25T14:59:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 39.491979] nix-grpc-daemon[786]: ts=2026-09-25T15:00:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.494818] nix-grpc-daemon[786]: ts=2026-09-25T15:00:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 39.695586] nix-grpc-daemon[786]: ts=2026-09-25T15:00:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.698519] nix-grpc-daemon[786]: ts=2026-09-25T15:00:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 39.899289] nix-grpc-daemon[786]: ts=2026-09-25T15:00:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.902223] nix-grpc-daemon[786]: ts=2026-09-25T15:00:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 40.104288] nix-grpc-daemon[786]: ts=2026-09-25T15:00:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 40.107244] nix-grpc-daemon[786]: ts=2026-09-25T15:00:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2 # [ 40.308001] nix-grpc-daemon[786]: ts=2026-09-25T15:00:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 40.310874] nix-grpc-daemon[786]: ts=2026-09-25T15:00:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 40.511652] nix-grpc-daemon[786]: ts=2026-09-25T15:00:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 40.514699] nix-grpc-daemon[786]: ts=2026-09-25T15:00:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 40.715399] nix-grpc-daemon[786]: ts=2026-09-25T15:00:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 40.718413] nix-grpc-daemon[786]: ts=2026-09-25T15:00:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 40.832803] nix-grpc-daemon[485]: E0000 00:00:1790348401.360061 539 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 40.856076] nix-grpc-daemon[485]: E0000 00:00:1790348401.388017 539 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 40.884000] nix-grpc-daemon[485]: E0000 00:00:1790348401.393262 539 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 40.887593] nix-grpc-daemon[485]: E0000 00:00:1790348401.414881 539 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 40.919230] nix-grpc-daemon[786]: ts=2026-09-25T15:00:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 40.922207] nix-grpc-daemon[786]: ts=2026-09-25T15:00:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 41.123936] nix-grpc-daemon[786]: ts=2026-09-25T15:00:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 41.126395] nix-grpc-daemon[786]: ts=2026-09-25T15:00:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2 # [ 41.324619] nix-grpc-daemon[786]: ts=2026-09-25T15:00:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 41.327449] nix-grpc-daemon[786]: ts=2026-09-25T15:00:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 41.540277] nix-grpc-daemon[485]: ts=2026-09-25T15:00:02Z level=info event=scheduler_take_over lb # [ 41.517243] niks3-server[1035]: 2026/09/25 15:00:02 INFO lead: acquired remote=192.168.1.3:45362 worker2 # [ 41.528220] nix-grpc-daemon[786]: ts=2026-09-25T15:00:02Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 41.531105] nix-grpc-daemon[786]: ts=2026-09-25T15:00:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 41.731750] nix-grpc-daemon[786]: ts=2026-09-25T15:00:02Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 41.734662] nix-grpc-daemon[786]: ts=2026-09-25T15:00:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 41.935433] nix-grpc-daemon[786]: ts=2026-09-25T15:00:02Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 41.938380] nix-grpc-daemon[786]: ts=2026-09-25T15:00:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 42.093604] nix-grpc-daemon[485]: ts=2026-09-25T15:00:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 42.096863] nix-grpc-daemon[485]: ts=2026-09-25T15:00:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 42.140197] nix-grpc-daemon[786]: ts=2026-09-25T15:00:02Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 42.142458] nix-grpc-daemon[786]: ts=2026-09-25T15:00:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 42.297400] nix-grpc-daemon[485]: ts=2026-09-25T15:00:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 42.300357] nix-grpc-daemon[485]: ts=2026-09-25T15:00:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2 # [ 42.340700] nix-grpc-daemon[786]: ts=2026-09-25T15:00:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 42.380171] nix-grpc-daemon[485]: ts=2026-09-25T15:00:02Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 42.383319] nix-grpc-daemon[485]: ts=2026-09-25T15:00:02Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:60128 worker1 # [ 42.386363] nix-grpc-daemon[485]: ts=2026-09-25T15:00:02Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 42.500856] nix-grpc-daemon[485]: ts=2026-09-25T15:00:03Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 42.504663] nix-grpc-daemon[485]: ts=2026-09-25T15:00:03Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:60128 worker1 # [ 42.507180] nix-grpc-daemon[485]: ts=2026-09-25T15:00:03Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1 # [ 52.749642] nix-grpc-daemon[485]: ts=2026-09-25T15:00:13Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:53772 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: (finished: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux, in 19.77 seconds) subtest: one node schedules and both builders hold a WorkerSession on it lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open' worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) (finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.47 seconds) subtest: balancer auth: client cert, bearer token, nothing, unknown CN worker2 # [ 54.392460] nix-grpc-daemon[786]: ts=2026-09-25T15:00:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 54.402635] systemd[1]: Started Nix Daemon. worker2 # [ 54.546214] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 54.566378] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 66 ms (attempt 1/5) worker2 # [ 54.636150] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 24 ms (attempt 2/5) worker2 # [ 54.661269] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 108 ms (attempt 3/5) worker2 # [ 54.774334] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 359 ms (attempt 4/5) worker2 # [ 55.138837] nix-grpc-daemon[786]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 55.142345] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 55.151260] nix-grpc-daemon[786]: ts=2026-09-25T15:00:15Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 55.158416] nix-grpc-daemon[786]: ts=2026-09-25T15:00:15Z level=debug event=path_info_miss path=jgg1bkcnnwpis3sdm7gl6kzqxlwc5ivn-nixos-system-client-test error="error: path '/nix/store/jgg1bkcnnwpis3sdm7gl6kzqxlwc5ivn-nixos-system-client-test' is required, but there is no substituter that can build it" worker2 # [ 55.165613] nix-grpc-daemon[786]: ts=2026-09-25T15:00:15Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 55.171717] nix-grpc-daemon[786]: ts=2026-09-25T15:00:15Z level=debug event=path_info_miss path=jgg1bkcnnwpis3sdm7gl6kzqxlwc5ivn-nixos-system-client-test error="error: path '/nix/store/jgg1bkcnnwpis3sdm7gl6kzqxlwc5ivn-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.05 seconds) client: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key client: (finished: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.03 seconds) worker2 # [ 55.427035] nix-grpc-daemon[786]: ts=2026-09-25T15:00:15Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 55.430903] nix-grpc-daemon[786]: ts=2026-09-25T15:00:15Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:50262 worker1 # [ 55.471810] nix-grpc-daemon[485]: ts=2026-09-25T15:00:16Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:53790 worker1 # [ 55.481228] systemd[1]: Started Nix Daemon. worker1 # [ 55.621510] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 55.640061] nix-grpc-daemon[485]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 76 ms (attempt 1/5) worker1 # [ 55.719690] nix-grpc-daemon[485]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 76 ms (attempt 2/5) worker1 # [ 55.796863] nix-grpc-daemon[485]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 233 ms (attempt 3/5) worker1 # [ 56.030855] nix-grpc-daemon[485]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 576 ms (attempt 4/5) worker1 # [ 56.609147] nix-grpc-daemon[485]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 56.611080] nix-grpc-daemon[485]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 56.619555] nix-grpc-daemon[485]: ts=2026-09-25T15:00:17Z level=debug event=path_info_miss path=jgg1bkcnnwpis3sdm7gl6kzqxlwc5ivn-nixos-system-client-test error="error: path '/nix/store/jgg1bkcnnwpis3sdm7gl6kzqxlwc5ivn-nixos-system-client-test' is required, but there is no substituter that can build it" worker1 # [ 56.627159] nix-grpc-daemon[485]: ts=2026-09-25T15:00:17Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:53790 worker1 # [ 56.634610] nix-grpc-daemon[485]: ts=2026-09-25T15:00:17Z level=debug event=path_info_miss path=jgg1bkcnnwpis3sdm7gl6kzqxlwc5ivn-nixos-system-client-test error="error: path '/nix/store/jgg1bkcnnwpis3sdm7gl6kzqxlwc5ivn-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: rm -r /var/lib/nix-grpc-store client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.02 seconds) worker2 # [ 56.808245] nix-grpc-daemon[786]: ts=2026-09-25T15:00:17Z level=info event=denied method=QueryMissing cn=- role=none worker2 # [ 56.996120] nix-grpc-daemon[786]: ts=2026-09-25T15:00:17Z level=info event=denied method=QueryMissing cn=stranger role=none (finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 2.82 seconds) subtest: DAG build is scheduled across workers and lands in the cache client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 worker2 # [ 57.163022] nix-grpc-daemon[786]: ts=2026-09-25T15:00:17Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:50262 client # don't know how to build these paths: client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker1 # [ 57.199214] nix-grpc-daemon[485]: ts=2026-09-25T15:00:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 57.223344] nix-grpc-daemon[485]: ts=2026-09-25T15:00:17Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=0 worker2 # [ 57.199100] nix-grpc-daemon[786]: ts=2026-09-25T15:00:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker1 # [ 57.226799] nix-grpc-daemon[485]: ts=2026-09-25T15:00:17Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 57.210112] nix-grpc-daemon[786]: ts=2026-09-25T15:00:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 client # copying 1 paths to 'lb:50051'... worker2 # [ 57.220531] nix-grpc-daemon[786]: ts=2026-09-25T15:00:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'... worker2 # [ 57.235858] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 57.260436] nix-grpc-daemon[786]: ts=2026-09-25T15:00:17Z level=info event=niks3_push_started pid=857 worker2 # [ 57.438512] nix-grpc-daemon[857]: time=2026-09-25T15:00:17.980Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 57.547624] nix-daemon[845]: accepted connection from pid 869, user nix-grpc-daemon (trusted) worker2 # [ 57.571361] nix-daemon[845]: reaped child process 872, status = succeeded lb # [ 57.576688] niks3-server[1035]: 2026/09/25 15:00:18 INFO Received push request method=POST path=/api/pushes worker2 # [ 57.584073] nix-grpc-daemon[857]: time=2026-09-25T15:00:18.125Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 57.585683] nix-grpc-daemon[857]: time=2026-09-25T15:00:18.125Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)" lb # [ 57.607734] niks3-server[1035]: 2026/09/25 15:00:18 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign lb # [ 57.646818] niks3-server[1035]: 2026/09/25 15:00:18 INFO Signed narinfos id=1 count=1 worker2 # [ 57.646351] nix-grpc-daemon[857]: time=2026-09-25T15:00:18.187Z level=INFO msg="Uploading 1 narinfos" lb # [ 57.652279] niks3-server[1035]: 2026/09/25 15:00:18 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls lb # [ 57.664952] niks3-server[1035]: 2026/09/25 15:00:18 INFO Received complete push request method=POST path=/api/pushes/1/complete lb # [ 57.677239] niks3-server[1035]: 2026/09/25 15:00:18 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst lb # [ 57.684570] niks3-server[1035]: 2026/09/25 15:00:18 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo worker2 # [ 57.684557] nix-grpc-daemon[857]: time=2026-09-25T15:00:18.227Z level=INFO msg="Upload complete. (242ms)" worker2 # [ 57.691488] nix-grpc-daemon[786]: ts=2026-09-25T15:00:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_in=504 worker2 # [ 57.702602] nix-grpc-daemon[786]: ts=2026-09-25T15:00:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 57.707990] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 57.712637] nix-daemon[845]: reaped child process 855, status = succeeded client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051... client # farm-a> worker2: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker2 # [ 57.724935] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds lb # [ 58.261512] envoy[897]: conn peer=[2001:db8:1::1]:56660 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=715 tx=361 ms=804 lb # [ 58.264185] envoy[897]: conn peer=[2001:db8:1::1]:56674 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2613 tx=406 ms=1197 lb # [ 58.266617] envoy[897]: conn peer=[2001:db8:1::1]:56676 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=7 lb # [ 58.268948] envoy[897]: conn peer=[2001:db8:1::1]:56680 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=12 client # farm-a> LOG-farm-a-t1 worker2 # [ 58.563132] nix-daemon[845]: reaped child process 877, status = succeeded worker2 # [ 58.672182] nix-daemon[845]: accepted connection from pid 893, user nix-grpc-daemon (trusted) worker2 # [ 58.692725] nix-daemon[845]: reaped child process 896, status = succeeded lb # [ 58.701262] niks3-server[1035]: 2026/09/25 15:00:19 INFO Received push request method=POST path=/api/pushes worker2 # [ 58.704518] nix-grpc-daemon[857]: time=2026-09-25T15:00:19.245Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 58.706040] nix-grpc-daemon[857]: time=2026-09-25T15:00:19.245Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)" lb # [ 58.726376] niks3-server[1035]: 2026/09/25 15:00:19 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign worker2 # [ 58.728397] nix-grpc-daemon[857]: time=2026-09-25T15:00:19.270Z level=INFO msg="Uploading 1 narinfos" lb # [ 58.738710] niks3-server[1035]: 2026/09/25 15:00:19 INFO Signed narinfos id=2 count=1 lb # [ 58.744501] niks3-server[1035]: 2026/09/25 15:00:19 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls lb # [ 58.747026] niks3-server[1035]: 2026/09/25 15:00:19 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv lb # [ 58.760576] niks3-server[1035]: 2026/09/25 15:00:19 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst lb # [ 58.763420] niks3-server[1035]: 2026/09/25 15:00:19 INFO Received complete push request method=POST path=/api/pushes/2/complete worker2 # [ 58.763834] nix-grpc-daemon[857]: time=2026-09-25T15:00:19.306Z level=INFO msg="Upload complete. (229ms)" lb # [ 58.768657] niks3-server[1035]: 2026/09/25 15:00:19 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo worker2 # [ 58.772809] nix-grpc-daemon[786]: ts=2026-09-25T15:00:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=1 worker1 # [ 58.804358] nix-grpc-daemon[485]: ts=2026-09-25T15:00:19Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=0 worker1 # [ 58.804558] nix-grpc-daemon[485]: ts=2026-09-25T15:00:19Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 58.783718] nix-grpc-daemon[786]: ts=2026-09-25T15:00:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 58.787904] nix-daemon[845]: reaped child process 875, status = succeeded worker2 # [ 58.793134] nix-grpc-daemon[786]: ts=2026-09-25T15:00:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 58.802193] nix-grpc-daemon[857]: time=2026-09-25T15:00:19.344Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 58.808850] nix-grpc-daemon[786]: ts=2026-09-25T15:00:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'... worker2 # [ 58.821218] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 58.945963] nix-daemon[845]: accepted connection from pid 900, user nix-grpc-daemon (trusted) worker2 # [ 58.969607] nix-daemon[845]: reaped child process 903, status = succeeded lb # [ 58.973845] niks3-server[1035]: 2026/09/25 15:00:19 INFO Received push request method=POST path=/api/pushes worker2 # [ 58.978816] nix-grpc-daemon[857]: time=2026-09-25T15:00:19.520Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 58.980693] nix-grpc-daemon[857]: time=2026-09-25T15:00:19.521Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)" lb # [ 58.994415] niks3-server[1035]: 2026/09/25 15:00:19 INFO Received sign narinfos request method=POST path=/api/pushes/3/sign worker2 # [ 58.993958] nix-grpc-daemon[857]: time=2026-09-25T15:00:19.536Z level=INFO msg="Uploading 1 narinfos" lb # [ 59.002147] niks3-server[1035]: 2026/09/25 15:00:19 INFO Signed narinfos id=3 count=1 lb # [ 59.007509] niks3-server[1035]: 2026/09/25 15:00:19 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst lb # [ 59.016815] niks3-server[1035]: 2026/09/25 15:00:19 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls lb # [ 59.019307] niks3-server[1035]: 2026/09/25 15:00:19 INFO Received complete push request method=POST path=/api/pushes/3/complete lb # [ 59.030862] niks3-server[1035]: 2026/09/25 15:00:19 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo worker2 # [ 59.032597] nix-grpc-daemon[857]: time=2026-09-25T15:00:19.575Z level=INFO msg="Upload complete. (202ms)" worker2 # [ 59.039059] nix-grpc-daemon[786]: ts=2026-09-25T15:00:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_in=624 worker2 # [ 59.050695] nix-grpc-daemon[786]: ts=2026-09-25T15:00:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 59.055721] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 59.059973] nix-daemon[845]: reaped child process 898, status = succeeded client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051... client # farm-b> worker2: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker2 # [ 59.072380] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-t1 worker2 # [ 59.907367] nix-daemon[845]: reaped child process 907, status = succeeded worker2 # [ 60.001073] nix-daemon[845]: accepted connection from pid 923, user nix-grpc-daemon (trusted) worker2 # [ 60.018078] nix-daemon[845]: reaped child process 926, status = succeeded lb # [ 60.027802] niks3-server[1035]: 2026/09/25 15:00:20 INFO Received push request method=POST path=/api/pushes worker2 # [ 60.030546] nix-grpc-daemon[857]: time=2026-09-25T15:00:20.572Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 60.032156] nix-grpc-daemon[857]: time=2026-09-25T15:00:20.572Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)" lb # [ 60.051539] niks3-server[1035]: 2026/09/25 15:00:20 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls worker2 # [ 60.054398] nix-grpc-daemon[857]: time=2026-09-25T15:00:20.596Z level=INFO msg="Uploading 1 narinfos" lb # [ 60.059655] niks3-server[1035]: 2026/09/25 15:00:20 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv lb # [ 60.066678] niks3-server[1035]: 2026/09/25 15:00:20 INFO Received sign narinfos request method=POST path=/api/pushes/4/sign lb # [ 60.071239] niks3-server[1035]: 2026/09/25 15:00:20 INFO Signed narinfos id=4 count=1 lb # [ 60.078027] niks3-server[1035]: 2026/09/25 15:00:20 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst lb # [ 60.084377] niks3-server[1035]: 2026/09/25 15:00:20 INFO Received complete push request method=POST path=/api/pushes/4/complete worker2 # [ 60.083411] nix-grpc-daemon[857]: time=2026-09-25T15:00:20.625Z level=INFO msg="Upload complete. (202ms)" lb # [ 60.088644] niks3-server[1035]: 2026/09/25 15:00:20 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo worker2 # [ 60.091440] nix-grpc-daemon[786]: ts=2026-09-25T15:00:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=1 worker1 # [ 60.121898] nix-grpc-daemon[485]: ts=2026-09-25T15:00:20Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=0 worker1 # [ 60.122086] nix-grpc-daemon[485]: ts=2026-09-25T15:00:20Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 60.102265] nix-daemon[845]: reaped child process 905, status = succeeded worker2 # [ 60.104773] nix-grpc-daemon[786]: ts=2026-09-25T15:00:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 60.114370] nix-grpc-daemon[786]: ts=2026-09-25T15:00:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 60.122942] nix-grpc-daemon[857]: time=2026-09-25T15:00:20.665Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 60.129675] nix-grpc-daemon[786]: ts=2026-09-25T15:00:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'... worker2 # [ 60.142072] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 60.262032] nix-daemon[845]: accepted connection from pid 930, user nix-grpc-daemon (trusted) worker2 # [ 60.281511] nix-daemon[845]: reaped child process 933, status = succeeded lb # [ 60.289087] niks3-server[1035]: 2026/09/25 15:00:20 INFO Received push request method=POST path=/api/pushes worker2 # [ 60.290306] nix-grpc-daemon[857]: time=2026-09-25T15:00:20.832Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 60.292563] nix-grpc-daemon[857]: time=2026-09-25T15:00:20.832Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)" lb # [ 60.306745] niks3-server[1035]: 2026/09/25 15:00:20 INFO Received sign narinfos request method=POST path=/api/pushes/5/sign worker2 # [ 60.307068] nix-grpc-daemon[857]: time=2026-09-25T15:00:20.849Z level=INFO msg="Uploading 1 narinfos" lb # [ 60.313287] niks3-server[1035]: 2026/09/25 15:00:20 INFO Signed narinfos id=5 count=1 lb # [ 60.319039] niks3-server[1035]: 2026/09/25 15:00:20 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls lb # [ 60.325409] niks3-server[1035]: 2026/09/25 15:00:20 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst lb # [ 60.334870] niks3-server[1035]: 2026/09/25 15:00:20 INFO Received complete push request method=POST path=/api/pushes/5/complete worker2 # [ 60.336585] nix-grpc-daemon[857]: time=2026-09-25T15:00:20.879Z level=INFO msg="Upload complete. (185ms)" lb # [ 60.340864] niks3-server[1035]: 2026/09/25 15:00:20 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo worker2 # [ 60.343613] nix-grpc-daemon[786]: ts=2026-09-25T15:00:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_in=752 worker2 # [ 60.353864] nix-grpc-daemon[786]: ts=2026-09-25T15:00:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 60.358823] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 60.363195] nix-daemon[845]: reaped child process 928, status = succeeded client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051... worker2 # [ 60.372568] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) client # farm-top> worker2: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1 worker2 # [ 61.804654] nix-daemon[845]: reaped child process 937, status = succeeded worker2 # [ 61.919899] nix-daemon[845]: accepted connection from pid 953, user nix-grpc-daemon (trusted) worker2 # [ 61.941995] nix-daemon[845]: reaped child process 956, status = succeeded lb # [ 61.945237] niks3-server[1035]: 2026/09/25 15:00:22 INFO Received push request method=POST path=/api/pushes worker2 # [ 61.947511] nix-grpc-daemon[857]: time=2026-09-25T15:00:22.488Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 61.950451] nix-grpc-daemon[857]: time=2026-09-25T15:00:22.488Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)" lb # [ 61.967990] niks3-server[1035]: 2026/09/25 15:00:22 INFO Received sign narinfos request method=POST path=/api/pushes/6/sign worker2 # [ 61.969460] nix-grpc-daemon[857]: time=2026-09-25T15:00:22.511Z level=INFO msg="Uploading 1 narinfos" lb # [ 61.978737] niks3-server[1035]: 2026/09/25 15:00:22 INFO Signed narinfos id=6 count=1 lb # [ 61.982780] niks3-server[1035]: 2026/09/25 15:00:22 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst lb # [ 61.989786] niks3-server[1035]: 2026/09/25 15:00:22 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls worker2 # [ 61.996900] nix-grpc-daemon[857]: time=2026-09-25T15:00:22.539Z level=INFO msg="Upload complete. (213ms)" lb # [ 61.997587] niks3-server[1035]: 2026/09/25 15:00:22 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv lb # [ 62.003678] niks3-server[1035]: 2026/09/25 15:00:22 INFO Received complete push request method=POST path=/api/pushes/6/complete lb # [ 62.007331] niks3-server[1035]: 2026/09/25 15:00:22 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo worker2 # [ 62.004620] nix-grpc-daemon[786]: ts=2026-09-25T15:00:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=1 worker2 # [ 62.015287] nix-daemon[845]: reaped child process 935, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 5.02 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.14 seconds) client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.02 seconds) client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client # copying 3 paths... client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'... client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'... client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.17 seconds) (finished: subtest: DAG build is scheduled across workers and lands in the cache, in 5.41 seconds) subtest: repeat is answered Cached by the scheduler without building worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker1 # finding garbage collector roots... worker1 # deleting specified paths... worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker1 # deleting unused links... worker1 # note: hard linking is currently saving 0.0 KiB worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.19 seconds) worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker2 # finding garbage collector roots... worker2 # deleting specified paths... worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker2 # deleting unused links... worker2 # note: hard linking is currently saving 0.0 KiB worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.18 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 worker1 # [ 62.994476] nix-grpc-daemon[485]: ts=2026-09-25T15:00:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53790 client # don't know how to build these paths: client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker2 # [ 62.975184] nix-grpc-daemon[786]: ts=2026-09-25T15:00:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 62.998991] nix-grpc-daemon[857]: time=2026-09-25T15:00:23.540Z level=INFO msg="All 2 paths already cached" worker1 # [ 63.040086] nix-grpc-daemon[485]: ts=2026-09-25T15:00:23Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=1 client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 0.25 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.14 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) (finished: subtest: repeat is answered Cached by the scheduler without building, in 0.91 seconds) subtest: inputs already in the cache are not sent to the scheduler lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.03 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2 worker1 # [ 63.632494] nix-grpc-daemon[485]: ts=2026-09-25T15:00:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53790 client # don't know how to build these paths: client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker2 # [ 63.611576] nix-grpc-daemon[786]: ts=2026-09-25T15:00:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 63.617811] nix-grpc-daemon[857]: time=2026-09-25T15:00:24.160Z level=INFO msg="All 2 paths already cached" worker1 # [ 63.655161] nix-grpc-daemon[485]: ts=2026-09-25T15:00:24Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=0 worker1 # [ 63.658622] nix-grpc-daemon[485]: ts=2026-09-25T15:00:24Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 63.633219] nix-grpc-daemon[786]: ts=2026-09-25T15:00:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 63.642873] nix-grpc-daemon[786]: ts=2026-09-25T15:00:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 63.652056] nix-grpc-daemon[857]: time=2026-09-25T15:00:24.194Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'... worker2 # [ 63.659073] nix-grpc-daemon[786]: ts=2026-09-25T15:00:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 63.668479] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 63.797215] nix-daemon[845]: accepted connection from pid 973, user nix-grpc-daemon (trusted) lb # [ 63.821162] niks3-server[1035]: 2026/09/25 15:00:24 INFO Received push request method=POST path=/api/pushes worker2 # [ 63.819489] nix-daemon[845]: reaped child process 976, status = succeeded worker2 # [ 63.823104] nix-grpc-daemon[857]: time=2026-09-25T15:00:24.365Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 63.824986] nix-grpc-daemon[857]: time=2026-09-25T15:00:24.365Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)" lb # [ 63.841618] niks3-server[1035]: 2026/09/25 15:00:24 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst worker2 # [ 63.843561] nix-grpc-daemon[857]: time=2026-09-25T15:00:24.385Z level=INFO msg="Uploading 1 narinfos" lb # [ 63.849369] niks3-server[1035]: 2026/09/25 15:00:24 INFO Received sign narinfos request method=POST path=/api/pushes/7/sign lb # [ 63.851729] niks3-server[1035]: 2026/09/25 15:00:24 INFO Signed narinfos id=7 count=1 lb # [ 63.859419] niks3-server[1035]: 2026/09/25 15:00:24 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls lb # [ 63.866274] niks3-server[1035]: 2026/09/25 15:00:24 INFO Received complete push request method=POST path=/api/pushes/7/complete worker2 # [ 63.869957] nix-grpc-daemon[857]: time=2026-09-25T15:00:24.412Z level=INFO msg="Upload complete. (191ms)" lb # [ 63.876182] niks3-server[1035]: 2026/09/25 15:00:24 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo worker2 # [ 63.876181] nix-grpc-daemon[786]: ts=2026-09-25T15:00:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_in=752 worker2 # [ 63.887848] nix-grpc-daemon[786]: ts=2026-09-25T15:00:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 63.893347] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 63.897964] nix-daemon[845]: reaped child process 971, status = succeeded client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051... client # farm-top> worker2: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker2 # [ 63.910405] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1b worker2 # [ 65.350964] nix-daemon[845]: reaped child process 980, status = succeeded worker2 # [ 65.464154] nix-daemon[845]: accepted connection from pid 996, user nix-grpc-daemon (trusted) lb # [ 65.487364] niks3-server[1035]: 2026/09/25 15:00:26 INFO Received push request method=POST path=/api/pushes worker2 # [ 65.485927] nix-daemon[845]: reaped child process 999, status = succeeded worker2 # [ 65.491932] nix-grpc-daemon[857]: time=2026-09-25T15:00:26.031Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 65.493889] nix-grpc-daemon[857]: time=2026-09-25T15:00:26.031Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)" lb # [ 65.514631] niks3-server[1035]: 2026/09/25 15:00:26 INFO Received sign narinfos request method=POST path=/api/pushes/8/sign worker2 # [ 65.518103] nix-grpc-daemon[857]: time=2026-09-25T15:00:26.060Z level=INFO msg="Uploading 1 narinfos" lb # [ 65.523015] niks3-server[1035]: 2026/09/25 15:00:26 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls lb # [ 65.529628] niks3-server[1035]: 2026/09/25 15:00:26 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst lb # [ 65.539835] niks3-server[1035]: 2026/09/25 15:00:26 INFO Signed narinfos id=8 count=1 lb # [ 65.544179] niks3-server[1035]: 2026/09/25 15:00:26 INFO Received complete push request method=POST path=/api/pushes/8/complete worker2 # [ 65.543667] nix-grpc-daemon[857]: time=2026-09-25T15:00:26.086Z level=INFO msg="Upload complete. (223ms)" lb # [ 65.548096] niks3-server[1035]: 2026/09/25 15:00:26 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv lb # [ 65.550874] niks3-server[1035]: 2026/09/25 15:00:26 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo worker2 # [ 65.552741] nix-grpc-daemon[786]: ts=2026-09-25T15:00:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=1 worker2 # [ 65.562973] nix-daemon[845]: reaped child process 978, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.15 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.15 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.03 seconds) (finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.50 seconds) subtest: outputs read back through any worker carry the cache signature client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2 client # don't know how to build these paths: client # /nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker2 # [ 66.011201] nix-grpc-daemon[786]: ts=2026-09-25T15:00:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:50262 worker1 # [ 66.042556] nix-grpc-daemon[485]: ts=2026-09-25T15:00:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 66.057626] nix-grpc-daemon[485]: ts=2026-09-25T15:00:26Z level=debug event=want client=4 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv cached=0 worker1 # [ 66.061663] nix-grpc-daemon[485]: ts=2026-09-25T15:00:26Z level=debug event=assigned drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 66.065730] nix-grpc-daemon[485]: ts=2026-09-25T15:00:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 66.075397] nix-grpc-daemon[485]: ts=2026-09-25T15:00:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53790 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' to 'lb:50051'... worker1 # [ 66.085213] nix-grpc-daemon[485]: ts=2026-09-25T15:00:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 66.094382] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 66.112618] nix-grpc-daemon[485]: ts=2026-09-25T15:00:26Z level=info event=niks3_push_started pid=905 worker1 # [ 66.261730] nix-grpc-daemon[905]: time=2026-09-25T15:00:26.793Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 66.367442] nix-daemon[855]: accepted connection from pid 917, user nix-grpc-daemon (trusted) lb # [ 66.367374] niks3-server[1035]: 2026/09/25 15:00:26 INFO Received push request method=POST path=/api/pushes worker1 # [ 66.393918] nix-daemon[855]: reaped child process 920, status = succeeded worker1 # [ 66.586749] nix-grpc-daemon[905]: time=2026-09-25T15:00:27.116Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 66.593340] nix-grpc-daemon[905]: time=2026-09-25T15:00:27.117Z level=INFO msg="Uploading 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv (512B)" lb # [ 66.583400] niks3-server[1035]: 2026/09/25 15:00:27 INFO Received sign narinfos request method=POST path=/api/pushes/9/sign worker1 # [ 66.613558] nix-grpc-daemon[905]: time=2026-09-25T15:00:27.144Z level=INFO msg="Uploading 1 narinfos" lb # [ 66.592419] niks3-server[1035]: 2026/09/25 15:00:27 INFO Signed narinfos id=9 count=1 lb # [ 66.594153] niks3-server[1035]: 2026/09/25 15:00:27 INFO Registered completed upload object_key=nar/13mhmmy2qay13imbbdi287vhryz1hwnyg0w6agd9s0ph37wnsqll.nar.zst lb # [ 66.603949] niks3-server[1035]: 2026/09/25 15:00:27 INFO Received complete push request method=POST path=/api/pushes/9/complete lb # [ 66.609941] niks3-server[1035]: 2026/09/25 15:00:27 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.ls lb # [ 66.615219] niks3-server[1035]: 2026/09/25 15:00:27 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.narinfo worker1 # [ 66.649048] nix-grpc-daemon[905]: time=2026-09-25T15:00:27.180Z level=INFO msg="Upload complete. (383ms)" worker1 # [ 66.656100] nix-grpc-daemon[485]: ts=2026-09-25T15:00:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 66.659334] nix-grpc-daemon[485]: ts=2026-09-25T15:00:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 66.671946] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 66.676582] nix-daemon[855]: reaped child process 903, status = succeeded client # building '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' on lb:50051... client # farm-a> node-a: building 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker1 # [ 66.688067] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-sig worker1 # [ 68.127442] nix-daemon[855]: reaped child process 924, status = succeeded worker1 # [ 68.240414] nix-daemon[855]: accepted connection from pid 940, user nix-grpc-daemon (trusted) lb # [ 68.235331] niks3-server[1035]: 2026/09/25 15:00:28 INFO Received push request method=POST path=/api/pushes worker1 # [ 68.261493] nix-daemon[855]: reaped child process 943, status = succeeded lb # [ 68.262756] envoy[897]: conn peer=[2001:db8:1::1]:56692 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9519 tx=4367 ms=4865 lb # [ 68.265647] envoy[897]: conn peer=[2001:db8:1::1]:54074 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=480 ms=60 lb # [ 68.268155] envoy[897]: conn peer=[2001:db8:1::1]:54080 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4168 tx=1902 ms=1958 worker1 # [ 68.399720] nix-grpc-daemon[905]: time=2026-09-25T15:00:28.928Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 68.404487] nix-grpc-daemon[905]: time=2026-09-25T15:00:28.928Z level=INFO msg="Uploading 2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a (128B)" lb # [ 68.395895] niks3-server[1035]: 2026/09/25 15:00:28 INFO Received sign narinfos request method=POST path=/api/pushes/10/sign worker1 # [ 68.423933] nix-grpc-daemon[905]: time=2026-09-25T15:00:28.955Z level=INFO msg="Uploading 1 narinfos" lb # [ 68.402349] niks3-server[1035]: 2026/09/25 15:00:28 INFO Signed narinfos id=10 count=1 lb # [ 68.409390] niks3-server[1035]: 2026/09/25 15:00:28 INFO Received complete push request method=POST path=/api/pushes/10/complete lb # [ 68.414913] niks3-server[1035]: 2026/09/25 15:00:28 INFO Registered completed upload object_key=log/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv lb # [ 68.421311] niks3-server[1035]: 2026/09/25 15:00:28 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst lb # [ 68.430256] niks3-server[1035]: 2026/09/25 15:00:28 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.ls worker1 # [ 68.463380] nix-grpc-daemon[905]: time=2026-09-25T15:00:28.995Z level=INFO msg="Upload complete. (363ms)" lb # [ 68.442467] niks3-server[1035]: 2026/09/25 15:00:28 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.narinfo worker1 # [ 68.470755] nix-grpc-daemon[485]: ts=2026-09-25T15:00:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=1 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv assign_id=5 outputs=1 worker1 # [ 68.483315] nix-grpc-daemon[485]: ts=2026-09-25T15:00:29Z level=debug event=want client=4 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv cached=0 worker1 # [ 68.491068] nix-grpc-daemon[485]: ts=2026-09-25T15:00:29Z level=debug event=assigned drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 68.495539] nix-grpc-daemon[485]: ts=2026-09-25T15:00:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 68.499938] nix-daemon[855]: reaped child process 922, status = succeeded worker1 # [ 68.502858] nix-grpc-daemon[485]: ts=2026-09-25T15:00:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 68.508568] nix-grpc-daemon[905]: time=2026-09-25T15:00:29.040Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' to 'lb:50051'... worker1 # [ 68.514887] nix-grpc-daemon[485]: ts=2026-09-25T15:00:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 68.527927] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 68.641735] nix-daemon[855]: accepted connection from pid 949, user nix-grpc-daemon (trusted) worker1 # [ 68.662243] nix-daemon[855]: reaped child process 952, status = succeeded lb # [ 68.641590] niks3-server[1035]: 2026/09/25 15:00:29 INFO Received push request method=POST path=/api/pushes worker1 # [ 68.671094] nix-grpc-daemon[905]: time=2026-09-25T15:00:29.202Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 68.672862] nix-grpc-daemon[905]: time=2026-09-25T15:00:29.202Z level=INFO msg="Uploading z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv (624B)" worker1 # [ 68.683696] nix-grpc-daemon[905]: time=2026-09-25T15:00:29.215Z level=INFO msg="Uploading 1 narinfos" lb # [ 68.659845] niks3-server[1035]: 2026/09/25 15:00:29 INFO Received sign narinfos request method=POST path=/api/pushes/11/sign lb # [ 68.667255] niks3-server[1035]: 2026/09/25 15:00:29 INFO Signed narinfos id=11 count=1 lb # [ 68.671824] niks3-server[1035]: 2026/09/25 15:00:29 INFO Registered completed upload object_key=nar/0cnhkgsbmxv5hj7zw3fa59qfxikw6251mlhz2gz1v6pgn8c8shwp.nar.zst lb # [ 68.680414] niks3-server[1035]: 2026/09/25 15:00:29 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.ls lb # [ 68.682929] niks3-server[1035]: 2026/09/25 15:00:29 INFO Received complete push request method=POST path=/api/pushes/11/complete worker1 # [ 68.712784] nix-grpc-daemon[905]: time=2026-09-25T15:00:29.244Z level=INFO msg="Upload complete. (180ms)" lb # [ 68.691426] niks3-server[1035]: 2026/09/25 15:00:29 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.narinfo worker1 # [ 68.720108] nix-grpc-daemon[485]: ts=2026-09-25T15:00:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 68.728588] nix-grpc-daemon[485]: ts=2026-09-25T15:00:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 68.733932] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 68.738136] nix-daemon[855]: reaped child process 947, status = succeeded client # building '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' on lb:50051... client # farm-b> node-a: building z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker1 # [ 68.747286] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-sig worker1 # [ 69.986878] nix-daemon[855]: reaped child process 956, status = succeeded worker1 # [ 70.086995] nix-daemon[855]: accepted connection from pid 972, user nix-grpc-daemon (trusted) lb # [ 70.079766] niks3-server[1035]: 2026/09/25 15:00:30 INFO Received push request method=POST path=/api/pushes worker1 # [ 70.105697] nix-daemon[855]: reaped child process 975, status = succeeded worker1 # [ 70.109957] nix-grpc-daemon[905]: time=2026-09-25T15:00:30.640Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 70.111412] nix-grpc-daemon[905]: time=2026-09-25T15:00:30.641Z level=INFO msg="Uploading h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b (176B)" lb # [ 70.099988] niks3-server[1035]: 2026/09/25 15:00:30 INFO Registered completed upload object_key=nar/0f75gpfkjj0g9xi2113yvvpdw0l16spc2f1xik1ifnzfbrvx4pfk.nar.zst worker1 # [ 70.133146] nix-grpc-daemon[905]: time=2026-09-25T15:00:30.665Z level=INFO msg="Uploading 1 narinfos" lb # [ 70.110052] niks3-server[1035]: 2026/09/25 15:00:30 INFO Received sign narinfos request method=POST path=/api/pushes/12/sign lb # [ 70.116487] niks3-server[1035]: 2026/09/25 15:00:30 INFO Signed narinfos id=12 count=1 lb # [ 70.120963] niks3-server[1035]: 2026/09/25 15:00:30 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.ls lb # [ 70.126702] niks3-server[1035]: 2026/09/25 15:00:30 INFO Registered completed upload object_key=log/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv lb # [ 70.135997] niks3-server[1035]: 2026/09/25 15:00:30 INFO Received complete push request method=POST path=/api/pushes/12/complete worker1 # [ 70.163147] nix-grpc-daemon[905]: time=2026-09-25T15:00:30.695Z level=INFO msg="Upload complete. (205ms)" lb # [ 70.140000] niks3-server[1035]: 2026/09/25 15:00:30 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.narinfo worker1 # [ 70.172497] nix-grpc-daemon[485]: ts=2026-09-25T15:00:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=1 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv assign_id=6 outputs=1 worker1 # [ 70.183701] nix-grpc-daemon[485]: ts=2026-09-25T15:00:30Z level=debug event=want client=4 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv cached=0 worker1 # [ 70.189186] nix-grpc-daemon[485]: ts=2026-09-25T15:00:30Z level=debug event=assigned drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 70.193113] nix-grpc-daemon[485]: ts=2026-09-25T15:00:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 70.195837] nix-daemon[855]: reaped child process 954, status = succeeded worker1 # [ 70.200679] nix-grpc-daemon[485]: ts=2026-09-25T15:00:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 70.208171] nix-grpc-daemon[905]: time=2026-09-25T15:00:30.740Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' to 'lb:50051'... worker1 # [ 70.214712] nix-grpc-daemon[485]: ts=2026-09-25T15:00:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 70.223782] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 70.332774] nix-daemon[855]: accepted connection from pid 981, user nix-grpc-daemon (trusted) worker1 # [ 70.351805] nix-daemon[855]: reaped child process 984, status = succeeded lb # [ 70.332254] niks3-server[1035]: 2026/09/25 15:00:30 INFO Received push request method=POST path=/api/pushes worker1 # [ 70.360845] nix-grpc-daemon[905]: time=2026-09-25T15:00:30.892Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 70.362355] nix-grpc-daemon[905]: time=2026-09-25T15:00:30.892Z level=INFO msg="Uploading knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv (752B)" lb # [ 70.350804] niks3-server[1035]: 2026/09/25 15:00:30 INFO Received sign narinfos request method=POST path=/api/pushes/13/sign worker1 # [ 70.380102] nix-grpc-daemon[905]: time=2026-09-25T15:00:30.911Z level=INFO msg="Uploading 1 narinfos" lb # [ 70.358974] niks3-server[1035]: 2026/09/25 15:00:30 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.ls lb # [ 70.365200] niks3-server[1035]: 2026/09/25 15:00:30 INFO Signed narinfos id=13 count=1 lb # [ 70.372967] niks3-server[1035]: 2026/09/25 15:00:30 INFO Registered completed upload object_key=nar/0lpagyrngav67x80d1hq0njjlkmqdkvqawraz8dsp36ag50v6iwf.nar.zst lb # [ 70.375921] niks3-server[1035]: 2026/09/25 15:00:30 INFO Received complete push request method=POST path=/api/pushes/13/complete worker1 # [ 70.407822] nix-grpc-daemon[905]: time=2026-09-25T15:00:30.939Z level=INFO msg="Upload complete. (176ms)" lb # [ 70.385346] niks3-server[1035]: 2026/09/25 15:00:30 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.narinfo worker1 # [ 70.414625] nix-grpc-daemon[485]: ts=2026-09-25T15:00:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 70.420627] nix-grpc-daemon[485]: ts=2026-09-25T15:00:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 70.429048] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 70.433358] nix-daemon[855]: reaped child process 979, status = succeeded client # building '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' on lb:50051... worker1 # [ 70.442739] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) client # farm-top> node-a: building knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-sig worker1 # [ 70.909992] nix-daemon[855]: reaped child process 988, status = succeeded worker1 # [ 70.972138] nix-daemon[855]: accepted connection from pid 1004, user nix-grpc-daemon (trusted) worker1 # [ 70.988494] nix-daemon[855]: reaped child process 1007, status = succeeded lb # [ 70.970265] niks3-server[1035]: 2026/09/25 15:00:31 INFO Received push request method=POST path=/api/pushes worker1 # [ 71.000118] nix-grpc-daemon[905]: time=2026-09-25T15:00:31.530Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 71.001615] nix-grpc-daemon[905]: time=2026-09-25T15:00:31.530Z level=INFO msg="Uploading 4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top (232B)" lb # [ 70.991538] niks3-server[1035]: 2026/09/25 15:00:31 INFO Registered completed upload object_key=log/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv lb # [ 71.000430] niks3-server[1035]: 2026/09/25 15:00:31 INFO Registered completed upload object_key=nar/07ar364rwqqniwi2nkdpqlgv7ws7702n5lqz703ackfmssrh40v8.nar.zst worker1 # [ 71.029207] nix-grpc-daemon[905]: time=2026-09-25T15:00:31.561Z level=INFO msg="Uploading 1 narinfos" lb # [ 71.008416] niks3-server[1035]: 2026/09/25 15:00:31 INFO Received sign narinfos request method=POST path=/api/pushes/14/sign lb # [ 71.014458] niks3-server[1035]: 2026/09/25 15:00:31 INFO Signed narinfos id=14 count=1 lb # [ 71.021165] niks3-server[1035]: 2026/09/25 15:00:31 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.ls lb # [ 71.023625] niks3-server[1035]: 2026/09/25 15:00:31 INFO Received complete push request method=POST path=/api/pushes/14/complete worker1 # [ 71.052879] nix-grpc-daemon[905]: time=2026-09-25T15:00:31.584Z level=INFO msg="Upload complete. (200ms)" lb # [ 71.030814] niks3-server[1035]: 2026/09/25 15:00:31 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.narinfo worker1 # [ 71.059960] nix-grpc-daemon[485]: ts=2026-09-25T15:00:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=0 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv assign_id=7 outputs=1 worker1 # [ 71.071317] nix-daemon[855]: reaped child process 986, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2, in 5.23 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath, in 0.14 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker1 # [ 71.344731] nix-grpc-daemon[485]: ts=2026-09-25T15:00:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:58714 worker1 # [ 71.349532] nix-grpc-daemon[485]: ts=2026-09-25T15:00:31Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:58714 client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.14 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker2 # [ 71.460772] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:57766 client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked): client # /nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a client # /nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b client # /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker2 # [ 71.477640] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:57766 worker2 # [ 71.482595] nix-grpc-daemon[786]: copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'http://lb:5751'... worker2 # [ 71.491552] nix-grpc-daemon[786]: copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'http://lb:5751'... worker2 # [ 71.499291] nix-grpc-daemon[786]: copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.19 seconds) client: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top client # finding garbage collector roots... client # removing stale temporary roots file "/nix/var/nix/temproots/855" client # removing stale temporary roots file "/nix/var/nix/temproots/814" client # removing stale temporary roots file "/nix/var/nix/temproots/868" client # removing stale temporary roots file "/nix/var/nix/temproots/891" client # removing stale temporary roots file "/nix/var/nix/temproots/835" client # removing stale temporary roots file "/nix/var/nix/temproots/799" client # removing stale temporary roots file "/nix/var/nix/temproots/847" client # removing stale temporary roots file "/nix/var/nix/temproots/876" client # deleting specified paths... client # deleting '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' client # deleting unused links... client # note: hard linking is currently saving 0.0 KiB worker1 # [ 71.843182] nix-grpc-daemon[485]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53790 worker2 # [ 71.821181] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 71.828770] nix-grpc-daemon[857]: time=2026-09-25T15:00:32.370Z level=INFO msg="All 1 paths already cached" worker2 # [ 71.833571] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 71.838668] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:50262 client # copying 3 paths... worker2 # [ 71.847468] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:50262 client # copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'grpc://lb:50051'... worker2 # [ 71.863498] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 71.867869] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_out=232 worker1 # [ 71.899923] nix-grpc-daemon[485]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 71.904529] nix-grpc-daemon[485]: ts=2026-09-25T15:00:32Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=0 paths=1 nar_bytes_out=176 worker2 # [ 71.880944] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 71.884638] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_out=128 client # copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'grpc://lb:50051'... client # copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'grpc://lb:50051'... client: (finished: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.39 seconds) (finished: subtest: outputs read back through any worker carry the cache signature, in 6.08 seconds) subtest: two clients wanting the same drv build it once client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup client # [ 71.951523] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup1.service; invocation ID: 9ce9d87ec96149ad880e84025925a71c client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.04 seconds) client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup worker1 # [ 72.136220] nix-grpc-daemon[485]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53790 client # [ 72.117886] nix[944]: don't know how to build these paths: client # [ 72.119684] nix[944]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker1 # [ 72.148122] nix-grpc-daemon[485]: ts=2026-09-25T15:00:32Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker1 # [ 72.152228] nix-grpc-daemon[485]: ts=2026-09-25T15:00:32Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 72.127624] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 72.135944] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 client # [ 72.148110] nix[944]: copying 1 paths to 'lb:50051'... client # [ 72.148976] nix[944]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'... worker2 # [ 72.143615] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 72.153144] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 72.285439] nix-daemon[845]: accepted connection from pid 1006, user nix-grpc-daemon (trusted) worker2 # [ 72.303823] nix-daemon[845]: reaped child process 1009, status = succeeded lb # [ 72.313794] niks3-server[1035]: 2026/09/25 15:00:32 INFO Received push request method=POST path=/api/pushes worker2 # [ 72.320114] nix-grpc-daemon[857]: time=2026-09-25T15:00:32.862Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 72.321806] nix-grpc-daemon[857]: time=2026-09-25T15:00:32.862Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)" lb # [ 72.339584] niks3-server[1035]: 2026/09/25 15:00:32 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst worker2 # [ 72.343577] nix-grpc-daemon[857]: time=2026-09-25T15:00:32.885Z level=INFO msg="Uploading 1 narinfos" lb # [ 72.348673] niks3-server[1035]: 2026/09/25 15:00:32 INFO Received sign narinfos request method=POST path=/api/pushes/15/sign lb # [ 72.350982] niks3-server[1035]: 2026/09/25 15:00:32 INFO Signed narinfos id=15 count=1 lb # [ 72.358670] niks3-server[1035]: 2026/09/25 15:00:32 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls lb # [ 72.366992] niks3-server[1035]: 2026/09/25 15:00:32 INFO Received complete push request method=POST path=/api/pushes/15/complete worker2 # [ 72.367590] nix-grpc-daemon[857]: time=2026-09-25T15:00:32.910Z level=INFO msg="Upload complete. (203ms)" lb # [ 72.372598] niks3-server[1035]: 2026/09/25 15:00:32 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo worker2 # [ 72.375963] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_in=432 worker2 # [ 72.386088] nix-grpc-daemon[786]: ts=2026-09-25T15:00:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 72.391127] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 72.395498] nix-daemon[845]: reaped child process 1004, status = succeeded client # [ 72.408739] nix[944]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... worker2 # [ 72.406087] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) client # [ 73.014200] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup2.service; invocation ID: 2f92b5f2d3f142f49d023d893e4620dd client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.07 seconds) worker1 # [ 73.240313] nix-grpc-daemon[485]: ts=2026-09-25T15:00:33Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53790 client # [ 73.222782] nix[960]: don't know how to build these paths: client # [ 73.224063] nix[960]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker1 # [ 73.253820] nix-grpc-daemon[485]: ts=2026-09-25T15:00:33Z level=debug event=want client=6 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker2 # [ 73.231589] nix-grpc-daemon[786]: ts=2026-09-25T15:00:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 client # [ 73.240836] nix[960]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... client: must succeed: sleep 2 client: (finished: must succeed: sleep 2, in 2.03 seconds) client: waiting for success: ! systemctl is-active dedup1 dedup2 worker2 # [ 76.448272] nix-daemon[845]: reaped child process 1013, status = succeeded worker2 # [ 76.550127] nix-daemon[845]: accepted connection from pid 1043, user nix-grpc-daemon (trusted) worker2 # [ 76.566422] nix-daemon[845]: reaped child process 1046, status = succeeded lb # [ 76.576640] niks3-server[1035]: 2026/09/25 15:00:37 INFO Received push request method=POST path=/api/pushes worker2 # [ 76.580462] nix-grpc-daemon[857]: time=2026-09-25T15:00:37.121Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 76.581992] nix-grpc-daemon[857]: time=2026-09-25T15:00:37.121Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)" lb # [ 76.603510] niks3-server[1035]: 2026/09/25 15:00:37 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls worker2 # [ 76.607812] nix-grpc-daemon[857]: time=2026-09-25T15:00:37.150Z level=INFO msg="Uploading 1 narinfos" lb # [ 76.611944] niks3-server[1035]: 2026/09/25 15:00:37 INFO Received sign narinfos request method=POST path=/api/pushes/16/sign lb # [ 76.617495] niks3-server[1035]: 2026/09/25 15:00:37 INFO Signed narinfos id=16 count=1 lb # [ 76.622182] niks3-server[1035]: 2026/09/25 15:00:37 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst lb # [ 76.629496] niks3-server[1035]: 2026/09/25 15:00:37 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv lb # [ 76.638568] niks3-server[1035]: 2026/09/25 15:00:37 INFO Received complete push request method=POST path=/api/pushes/16/complete worker2 # [ 76.639257] nix-grpc-daemon[857]: time=2026-09-25T15:00:37.181Z level=INFO msg="Upload complete. (222ms)" lb # [ 76.644279] niks3-server[1035]: 2026/09/25 15:00:37 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo worker2 # [ 76.648694] nix-grpc-daemon[786]: ts=2026-09-25T15:00:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker2 # [ 76.657839] nix-grpc-daemon[786]: ts=2026-09-25T15:00:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker2 # [ 76.662291] nix-daemon[845]: reaped child process 1011, status = succeeded client # [ 76.691668] systemd[1]: dedup1.service: Deactivated successfully. client # [ 76.694361] systemd[1]: dedup1.service: Consumed 149ms CPU time over 4.742s wall clock time, 41.5M memory peak, 7.1K incoming IP traffic, 8.3K outgoing IP traffic. client # [ 76.701776] systemd[1]: dedup2.service: Deactivated successfully. client # [ 76.703191] systemd[1]: dedup2.service: Consumed 185ms CPU time over 3.686s wall clock time, 35.2M memory peak, 5.9K incoming IP traffic, 5.6K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.07 seconds) client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; } client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.05 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.04 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) (finished: subtest: two clients wanting the same drv build it once, in 5.69 seconds) subtest: --timeout and --max-silent-time reach the builder worker2 # [ 77.785264] nix-grpc-daemon[786]: ts=2026-09-25T15:00:38Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:50262 worker1 # [ 77.823995] nix-grpc-daemon[485]: ts=2026-09-25T15:00:38Z level=debug event=want client=7 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv cached=0 worker1 # [ 77.828624] nix-grpc-daemon[485]: ts=2026-09-25T15:00:38Z level=debug event=assigned drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 77.833257] nix-grpc-daemon[485]: ts=2026-09-25T15:00:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 77.837273] nix-grpc-daemon[485]: ts=2026-09-25T15:00:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 77.845068] nix-grpc-daemon[485]: ts=2026-09-25T15:00:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 77.853648] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 77.980296] nix-daemon[855]: accepted connection from pid 1038, user nix-grpc-daemon (trusted) worker1 # [ 77.995648] nix-daemon[855]: reaped child process 1041, status = succeeded lb # [ 77.979543] niks3-server[1035]: 2026/09/25 15:00:38 INFO Received push request method=POST path=/api/pushes worker1 # [ 78.008727] nix-grpc-daemon[905]: time=2026-09-25T15:00:38.540Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 78.010347] nix-grpc-daemon[905]: time=2026-09-25T15:00:38.540Z level=INFO msg="Uploading sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv (424B)" lb # [ 77.993512] niks3-server[1035]: 2026/09/25 15:00:38 INFO Received sign narinfos request method=POST path=/api/pushes/17/sign worker1 # [ 78.022272] nix-grpc-daemon[905]: time=2026-09-25T15:00:38.554Z level=INFO msg="Uploading 1 narinfos" lb # [ 78.000538] niks3-server[1035]: 2026/09/25 15:00:38 INFO Signed narinfos id=17 count=1 lb # [ 78.005207] niks3-server[1035]: 2026/09/25 15:00:38 INFO Registered completed upload object_key=nar/1m12xmj6yzmb9y2j8ly0yvh5fyzy1nqk7ajhqwr4wn3b5h0bsch5.nar.zst lb # [ 78.011792] niks3-server[1035]: 2026/09/25 15:00:38 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.ls lb # [ 78.020690] niks3-server[1035]: 2026/09/25 15:00:38 INFO Received complete push request method=POST path=/api/pushes/17/complete lb # [ 78.024455] niks3-server[1035]: 2026/09/25 15:00:38 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.narinfo worker1 # [ 78.051779] nix-grpc-daemon[905]: time=2026-09-25T15:00:38.583Z level=INFO msg="Upload complete. (188ms)" worker1 # [ 78.057582] nix-grpc-daemon[485]: ts=2026-09-25T15:00:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=0 paths=1 nar_bytes_in=424 worker1 # [ 78.064757] nix-grpc-daemon[485]: ts=2026-09-25T15:00:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 78.073422] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 78.077370] nix-daemon[855]: reaped child process 1036, status = succeeded worker1 # [ 78.086087] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) lb # [ 78.261084] envoy[897]: conn peer=[2001:db8:1::1]:54084 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9511 tx=4329 ms=5032 lb # [ 78.263635] envoy[897]: conn peer=[2001:db8:1::1]:33380 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1034 tx=2868 ms=88 lb # [ 78.266364] envoy[897]: conn peer=[2001:db8:1::1]:33402 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=329 ms=24 lb # [ 78.268980] envoy[897]: conn peer=[2001:db8:1::1]:33388 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=384 tx=409 ms=41 lb # [ 78.271577] envoy[897]: conn peer=[2001:db8:1::1]:33396 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=375 ms=33 lb # [ 78.274354] envoy[897]: conn peer=[2001:db8:1::1]:33404 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2988 tx=1521 ms=4551 lb # [ 78.277555] envoy[897]: conn peer=[2001:db8:1::1]:33418 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1443 tx=1194 ms=3454 worker1 # [ 84.301593] nix-grpc-daemon[485]: ts=2026-09-25T15:00:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=6 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv assign_id=9 outputs=0 worker1 # [ 84.327770] nix-daemon[855]: reaped child process 1043, status = succeeded worker1 # [ 84.329989] nix-daemon[855]: reaped child process 1045, status = succeeded worker1 # [ 84.494222] nix-grpc-daemon[485]: ts=2026-09-25T15:00:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 84.503963] nix-grpc-daemon[485]: ts=2026-09-25T15:00:45Z level=debug event=want client=8 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv cached=0 worker2 # [ 84.479333] nix-grpc-daemon[786]: ts=2026-09-25T15:00:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker1 # [ 84.507064] nix-grpc-daemon[485]: ts=2026-09-25T15:00:45Z level=debug event=assigned drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 84.489538] nix-grpc-daemon[786]: ts=2026-09-25T15:00:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 84.497930] nix-grpc-daemon[786]: ts=2026-09-25T15:00:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 84.506736] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 84.636565] nix-daemon[845]: accepted connection from pid 1059, user nix-grpc-daemon (trusted) worker2 # [ 84.653398] nix-daemon[845]: reaped child process 1062, status = succeeded lb # [ 84.663247] niks3-server[1035]: 2026/09/25 15:00:45 INFO Received push request method=POST path=/api/pushes worker2 # [ 84.666341] nix-grpc-daemon[857]: time=2026-09-25T15:00:45.208Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 84.667875] nix-grpc-daemon[857]: time=2026-09-25T15:00:45.208Z level=INFO msg="Uploading pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv (424B)" lb # [ 84.684565] niks3-server[1035]: 2026/09/25 15:00:45 INFO Registered completed upload object_key=nar/14f5yrnllfav3svmw4v4svw15r4zlf5yv1p972jy14mmq6gdw943.nar.zst worker2 # [ 84.690414] nix-grpc-daemon[857]: time=2026-09-25T15:00:45.232Z level=INFO msg="Uploading 1 narinfos" lb # [ 84.694630] niks3-server[1035]: 2026/09/25 15:00:45 INFO Received sign narinfos request method=POST path=/api/pushes/18/sign lb # [ 84.700827] niks3-server[1035]: 2026/09/25 15:00:45 INFO Signed narinfos id=18 count=1 lb # [ 84.704648] niks3-server[1035]: 2026/09/25 15:00:45 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.ls lb # [ 84.716649] niks3-server[1035]: 2026/09/25 15:00:45 INFO Received complete push request method=POST path=/api/pushes/18/complete worker2 # [ 84.718819] nix-grpc-daemon[857]: time=2026-09-25T15:00:45.261Z level=INFO msg="Upload complete. (202ms)" lb # [ 84.722995] niks3-server[1035]: 2026/09/25 15:00:45 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.narinfo worker2 # [ 84.725685] nix-grpc-daemon[786]: ts=2026-09-25T15:00:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 84.732657] nix-grpc-daemon[786]: ts=2026-09-25T15:00:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 84.743554] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 84.745844] nix-daemon[845]: reaped child process 1057, status = succeeded worker2 # [ 84.757229] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) lb # [ 88.263124] envoy[897]: conn peer=[2001:db8:1::1]:33420 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2971 tx=956 ms=6504 worker2 # [ 90.973372] nix-grpc-daemon[786]: ts=2026-09-25T15:00:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=6 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv assign_id=10 outputs=0 worker2 # [ 90.990176] nix-daemon[845]: reaped child process 1064, status = succeeded worker2 # [ 90.991345] nix-daemon[845]: reaped child process 1066, status = succeeded (finished: subtest: --timeout and --max-silent-time reach the builder, in 13.46 seconds) subtest: the first client leaving does not fail the second client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share client # [ 91.115155] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share1.service; invocation ID: 59d8cb92c43c496d8db79f95ef426c06 client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 0.05 seconds) client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share worker2 # [ 91.297344] nix-grpc-daemon[786]: ts=2026-09-25T15:00:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:50262 client # [ 91.306729] nix[1032]: don't know how to build these paths: client # [ 91.308470] nix[1032]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker1 # [ 91.337444] nix-grpc-daemon[485]: ts=2026-09-25T15:00:51Z level=debug event=want client=9 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 worker1 # [ 91.341581] nix-grpc-daemon[485]: ts=2026-09-25T15:00:51Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 91.345836] nix-grpc-daemon[485]: ts=2026-09-25T15:00:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 91.351957] nix-grpc-daemon[485]: ts=2026-09-25T15:00:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53790 client # [ 91.337795] nix[1032]: copying 1 paths to 'lb:50051'... worker1 # [ 91.360005] nix-grpc-daemon[485]: ts=2026-09-25T15:00:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 client # [ 91.338934] nix[1032]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'... worker1 # [ 91.368749] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 91.497135] nix-daemon[855]: accepted connection from pid 1072, user nix-grpc-daemon (trusted) worker1 # [ 91.516089] nix-daemon[855]: reaped child process 1075, status = succeeded lb # [ 91.491050] niks3-server[1035]: 2026/09/25 15:00:52 INFO Received push request method=POST path=/api/pushes worker1 # [ 91.520373] nix-grpc-daemon[905]: time=2026-09-25T15:00:52.051Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 91.522109] nix-grpc-daemon[905]: time=2026-09-25T15:00:52.051Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)" lb # [ 91.505533] niks3-server[1035]: 2026/09/25 15:00:52 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst lb # [ 91.514047] niks3-server[1035]: 2026/09/25 15:00:52 INFO Received sign narinfos request method=POST path=/api/pushes/19/sign worker1 # [ 91.540967] nix-grpc-daemon[905]: time=2026-09-25T15:00:52.072Z level=INFO msg="Uploading 1 narinfos" lb # [ 91.521346] niks3-server[1035]: 2026/09/25 15:00:52 INFO Signed narinfos id=19 count=1 lb # [ 91.528676] niks3-server[1035]: 2026/09/25 15:00:52 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls lb # [ 91.540068] niks3-server[1035]: 2026/09/25 15:00:52 INFO Received complete push request method=POST path=/api/pushes/19/complete worker1 # [ 91.568033] nix-grpc-daemon[905]: time=2026-09-25T15:00:52.100Z level=INFO msg="Upload complete. (190ms)" lb # [ 91.546597] niks3-server[1035]: 2026/09/25 15:00:52 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo worker1 # [ 91.575753] nix-grpc-daemon[485]: ts=2026-09-25T15:00:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=0 paths=1 nar_bytes_in=432 worker1 # [ 91.584994] nix-grpc-daemon[485]: ts=2026-09-25T15:00:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 91.589555] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 91.596271] nix-daemon[855]: reaped child process 1070, status = succeeded client # [ 91.579309] nix[1032]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... worker1 # [ 91.604006] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) client # [ 92.192836] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share2.service; invocation ID: 619300efc577488295a2f37193cf055f client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 1.08 seconds) worker2 # [ 92.374460] nix-grpc-daemon[786]: ts=2026-09-25T15:00:52Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:50262 client # [ 92.384247] nix[1048]: don't know how to build these paths: client # [ 92.385998] nix[1048]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker1 # [ 92.416008] nix-grpc-daemon[485]: ts=2026-09-25T15:00:52Z level=debug event=want client=10 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 worker1 # [ 92.420624] nix-grpc-daemon[485]: ts=2026-09-25T15:00:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 client # [ 92.402686] nix[1048]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... client: must succeed: sleep 2; systemctl kill -s INT share1 client # [ 95.427549] systemd[1]: share1.service: Sent signal SIGINT to main process 1032 (nix) on client request. client # [ 95.431183] nix[1032]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client # [ 95.432567] nix[1032]: error: interrupted by the user client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.05 seconds) client: waiting for success: ! systemctl is-active share1 client # [ 95.485850] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE client # [ 95.488439] systemd[1]: share1.service: Failed with result 'exit-code'. client # [ 95.491376] systemd[1]: share1.service: Consumed 161ms CPU time over 4.373s wall clock time, 41.4M memory peak, 6.1K incoming IP traffic, 8.1K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share1, in 1.08 seconds) client: waiting for success: ! systemctl is-active share2 worker1 # [ 96.838413] nix-daemon[855]: reaped child process 1079, status = succeeded worker1 # [ 96.949083] nix-daemon[855]: accepted connection from pid 1115, user nix-grpc-daemon (trusted) worker1 # [ 96.969328] nix-daemon[855]: reaped child process 1118, status = succeeded lb # [ 96.948847] niks3-server[1035]: 2026/09/25 15:00:57 INFO Received push request method=POST path=/api/pushes worker1 # [ 96.980464] nix-grpc-daemon[905]: time=2026-09-25T15:00:57.511Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 96.981966] nix-grpc-daemon[905]: time=2026-09-25T15:00:57.511Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)" lb # [ 96.979383] niks3-server[1035]: 2026/09/25 15:00:57 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls worker1 # [ 97.010169] nix-grpc-daemon[905]: time=2026-09-25T15:00:57.542Z level=INFO msg="Uploading 1 narinfos" lb # [ 96.986993] niks3-server[1035]: 2026/09/25 15:00:57 INFO Received sign narinfos request method=POST path=/api/pushes/20/sign lb # [ 96.993511] niks3-server[1035]: 2026/09/25 15:00:57 INFO Signed narinfos id=20 count=1 lb # [ 96.995216] niks3-server[1035]: 2026/09/25 15:00:57 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst lb # [ 97.007739] niks3-server[1035]: 2026/09/25 15:00:57 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker1 # [ 97.038096] nix-grpc-daemon[905]: time=2026-09-25T15:00:57.570Z level=INFO msg="Upload complete. (222ms)" lb # [ 97.014706] niks3-server[1035]: 2026/09/25 15:00:57 INFO Received complete push request method=POST path=/api/pushes/20/complete lb # [ 97.017893] niks3-server[1035]: 2026/09/25 15:00:57 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo worker1 # [ 97.044724] nix-grpc-daemon[485]: ts=2026-09-25T15:00:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=4 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 worker1 # [ 97.057286] nix-grpc-daemon[485]: ts=2026-09-25T15:00:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=5 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 worker1 # [ 97.061619] nix-daemon[855]: reaped child process 1077, status = succeeded client # [ 97.053318] systemd[1]: share2.service: Deactivated successfully. client # [ 97.056218] systemd[1]: share2.service: Consumed 167ms CPU time over 4.862s wall clock time, 33.3M memory peak, 6K incoming IP traffic, 5.6K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share2, in 1.07 seconds) client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; } client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.04 seconds) (finished: subtest: the first client leaving does not fail the second, in 6.72 seconds) subtest: build hook: nix-daemon with builders = grpc://lb client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log client # [ 97.922123] systemd[1]: Started Nix Daemon. client # [ 98.030088] nix-daemon[1100]: accepted connection from pid 1097, user root (trusted) worker1 # [ 98.191539] nix-grpc-daemon[485]: ts=2026-09-25T15:00:58Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 98.195376] nix-grpc-daemon[485]: ts=2026-09-25T15:00:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 98.204097] nix-grpc-daemon[485]: ts=2026-09-25T15:00:58Z level=debug event=want client=11 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=0 worker1 # [ 98.206988] nix-grpc-daemon[485]: ts=2026-09-25T15:00:58Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 98.186184] nix-grpc-daemon[786]: ts=2026-09-25T15:00:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 98.195208] nix-grpc-daemon[786]: ts=2026-09-25T15:00:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 98.203327] nix-grpc-daemon[786]: ts=2026-09-25T15:00:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 98.212414] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) lb # [ 98.264361] envoy[897]: conn peer=[2001:db8:1::1]:44874 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2972 tx=934 ms=6519 lb # [ 98.267014] envoy[897]: conn peer=[2001:db8:1::1]:60214 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2973 tx=769 ms=4143 lb # [ 98.270571] envoy[897]: conn peer=[2001:db8:1::1]:60228 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1197 ms=4662 worker2 # [ 98.341246] nix-daemon[845]: accepted connection from pid 1105, user nix-grpc-daemon (trusted) worker2 # [ 98.358077] nix-daemon[845]: reaped child process 1108, status = succeeded lb # [ 98.366539] niks3-server[1035]: 2026/09/25 15:00:58 INFO Received push request method=POST path=/api/pushes worker2 # [ 98.368984] nix-grpc-daemon[857]: time=2026-09-25T15:00:58.910Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 98.370522] nix-grpc-daemon[857]: time=2026-09-25T15:00:58.910Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)" lb # [ 98.384325] niks3-server[1035]: 2026/09/25 15:00:58 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst worker2 # [ 98.391655] nix-grpc-daemon[857]: time=2026-09-25T15:00:58.934Z level=INFO msg="Uploading 1 narinfos" lb # [ 98.390869] niks3-server[1035]: 2026/09/25 15:00:58 INFO Received sign narinfos request method=POST path=/api/pushes/21/sign lb # [ 98.399034] niks3-server[1035]: 2026/09/25 15:00:58 INFO Signed narinfos id=21 count=1 lb # [ 98.402357] niks3-server[1035]: 2026/09/25 15:00:58 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls lb # [ 98.410013] niks3-server[1035]: 2026/09/25 15:00:58 INFO Received complete push request method=POST path=/api/pushes/21/complete worker2 # [ 98.414028] nix-grpc-daemon[857]: time=2026-09-25T15:00:58.956Z level=INFO msg="Upload complete. (192ms)" lb # [ 98.418512] niks3-server[1035]: 2026/09/25 15:00:58 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo worker2 # [ 98.420191] nix-grpc-daemon[786]: ts=2026-09-25T15:00:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 98.423547] nix-grpc-daemon[786]: ts=2026-09-25T15:00:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 98.437147] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 98.439575] nix-daemon[845]: reaped child process 1103, status = succeeded worker2 # [ 98.451034] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 99.921975] nix-daemon[845]: reaped child process 1112, status = succeeded worker2 # [ 99.980929] nix-daemon[845]: accepted connection from pid 1128, user nix-grpc-daemon (trusted) worker2 # [ 99.998884] nix-daemon[845]: reaped child process 1131, status = succeeded lb # [ 100.007419] niks3-server[1035]: 2026/09/25 15:01:00 INFO Received push request method=POST path=/api/pushes worker2 # [ 100.010421] nix-grpc-daemon[857]: time=2026-09-25T15:01:00.551Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 100.012101] nix-grpc-daemon[857]: time=2026-09-25T15:01:00.551Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)" worker2 # [ 100.032521] nix-grpc-daemon[857]: time=2026-09-25T15:01:00.574Z level=INFO msg="Uploading 1 narinfos" lb # [ 100.030947] niks3-server[1035]: 2026/09/25 15:01:00 INFO Received sign narinfos request method=POST path=/api/pushes/22/sign lb # [ 100.039721] niks3-server[1035]: 2026/09/25 15:01:00 INFO Signed narinfos id=22 count=1 lb # [ 100.044679] niks3-server[1035]: 2026/09/25 15:01:00 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls lb # [ 100.047135] niks3-server[1035]: 2026/09/25 15:01:00 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv lb # [ 100.059530] niks3-server[1035]: 2026/09/25 15:01:00 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst lb # [ 100.064075] niks3-server[1035]: 2026/09/25 15:01:00 INFO Received complete push request method=POST path=/api/pushes/22/complete worker2 # [ 100.064113] nix-grpc-daemon[857]: time=2026-09-25T15:01:00.606Z level=INFO msg="Upload complete. (204ms)" lb # [ 100.069917] niks3-server[1035]: 2026/09/25 15:01:00 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo worker2 # [ 100.072587] nix-grpc-daemon[786]: ts=2026-09-25T15:01:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=12 outputs=1 worker2 # [ 100.082731] nix-grpc-daemon[786]: ts=2026-09-25T15:01:00Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 100.086345] nix-daemon[845]: reaped child process 1110, status = succeeded worker2 # [ 100.098751] nix-grpc-daemon[786]: ts=2026-09-25T15:01:00Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 100.102712] nix-grpc-daemon[786]: ts=2026-09-25T15:01:00Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_out=128 worker2 # [ 100.301326] nix-grpc-daemon[786]: ts=2026-09-25T15:01:00Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 100.305619] nix-grpc-daemon[786]: ts=2026-09-25T15:01:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 100.309672] nix-grpc-daemon[857]: time=2026-09-25T15:01:00.851Z level=INFO msg="All 1 paths already cached" worker1 # [ 100.345792] nix-grpc-daemon[485]: ts=2026-09-25T15:01:00Z level=debug event=want client=12 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=0 worker2 # [ 100.319707] nix-grpc-daemon[786]: ts=2026-09-25T15:01:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker1 # [ 100.346011] nix-grpc-daemon[485]: ts=2026-09-25T15:01:00Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 100.328552] nix-grpc-daemon[786]: ts=2026-09-25T15:01:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 100.336190] nix-grpc-daemon[857]: time=2026-09-25T15:01:00.878Z level=INFO msg="All 1 paths already cached" worker2 # [ 100.341578] nix-grpc-daemon[786]: ts=2026-09-25T15:01:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 100.349438] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 100.474901] nix-daemon[845]: accepted connection from pid 1135, user nix-grpc-daemon (trusted) worker2 # [ 100.492839] nix-daemon[845]: reaped child process 1138, status = succeeded lb # [ 100.502258] niks3-server[1035]: 2026/09/25 15:01:01 INFO Received push request method=POST path=/api/pushes worker2 # [ 100.503375] nix-grpc-daemon[857]: time=2026-09-25T15:01:01.045Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 100.505063] nix-grpc-daemon[857]: time=2026-09-25T15:01:01.045Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)" lb # [ 100.517661] niks3-server[1035]: 2026/09/25 15:01:01 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst worker2 # [ 100.522647] nix-grpc-daemon[857]: time=2026-09-25T15:01:01.065Z level=INFO msg="Uploading 1 narinfos" lb # [ 100.527851] niks3-server[1035]: 2026/09/25 15:01:01 INFO Received sign narinfos request method=POST path=/api/pushes/23/sign lb # [ 100.533673] niks3-server[1035]: 2026/09/25 15:01:01 INFO Signed narinfos id=23 count=1 lb # [ 100.535413] niks3-server[1035]: 2026/09/25 15:01:01 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls lb # [ 100.543530] niks3-server[1035]: 2026/09/25 15:01:01 INFO Received complete push request method=POST path=/api/pushes/23/complete lb # [ 100.550118] niks3-server[1035]: 2026/09/25 15:01:01 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo worker2 # [ 100.549359] nix-grpc-daemon[857]: time=2026-09-25T15:01:01.091Z level=INFO msg="Upload complete. (190ms)" worker2 # [ 100.555801] nix-grpc-daemon[786]: ts=2026-09-25T15:01:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_in=624 worker2 # [ 100.567191] nix-grpc-daemon[786]: ts=2026-09-25T15:01:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 100.571819] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 100.575673] nix-daemon[845]: reaped child process 1133, status = succeeded worker2 # [ 100.584339] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 101.261564] nix-daemon[845]: reaped child process 1142, status = succeeded worker2 # [ 101.318939] nix-daemon[845]: accepted connection from pid 1158, user nix-grpc-daemon (trusted) worker2 # [ 101.337444] nix-daemon[845]: reaped child process 1161, status = succeeded lb # [ 101.345502] niks3-server[1035]: 2026/09/25 15:01:01 INFO Received push request method=POST path=/api/pushes worker2 # [ 101.348379] nix-grpc-daemon[857]: time=2026-09-25T15:01:01.889Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 101.349918] nix-grpc-daemon[857]: time=2026-09-25T15:01:01.889Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)" lb # [ 101.375872] niks3-server[1035]: 2026/09/25 15:01:01 INFO Received sign narinfos request method=POST path=/api/pushes/24/sign worker2 # [ 101.378281] nix-grpc-daemon[857]: time=2026-09-25T15:01:01.920Z level=INFO msg="Uploading 1 narinfos" lb # [ 101.384221] niks3-server[1035]: 2026/09/25 15:01:01 INFO Signed narinfos id=24 count=1 lb # [ 101.388218] niks3-server[1035]: 2026/09/25 15:01:01 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst lb # [ 101.396675] niks3-server[1035]: 2026/09/25 15:01:01 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls worker2 # [ 101.398138] nix-grpc-daemon[857]: time=2026-09-25T15:01:01.940Z level=INFO msg="Upload complete. (204ms)" lb # [ 101.402952] niks3-server[1035]: 2026/09/25 15:01:01 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv lb # [ 101.407977] niks3-server[1035]: 2026/09/25 15:01:01 INFO Received complete push request method=POST path=/api/pushes/24/complete worker2 # [ 101.404668] nix-grpc-daemon[786]: ts=2026-09-25T15:01:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=13 outputs=1 lb # [ 101.411155] niks3-server[1035]: 2026/09/25 15:01:01 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo worker2 # [ 101.417455] nix-grpc-daemon[786]: ts=2026-09-25T15:01:01Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 101.420961] nix-daemon[845]: reaped child process 1140, status = succeeded worker2 # [ 101.433333] nix-grpc-daemon[786]: ts=2026-09-25T15:01:01Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 101.437076] nix-grpc-daemon[786]: ts=2026-09-25T15:01:01Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_out=176 worker2 # [ 101.630469] nix-grpc-daemon[786]: ts=2026-09-25T15:01:02Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 101.634723] nix-grpc-daemon[786]: ts=2026-09-25T15:01:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 101.638985] nix-grpc-daemon[857]: time=2026-09-25T15:01:02.180Z level=INFO msg="All 2 paths already cached" worker1 # [ 101.674709] nix-grpc-daemon[485]: ts=2026-09-25T15:01:02Z level=debug event=want client=13 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=0 worker1 # [ 101.678272] nix-grpc-daemon[485]: ts=2026-09-25T15:01:02Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 101.652812] nix-grpc-daemon[786]: ts=2026-09-25T15:01:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 101.661667] nix-grpc-daemon[786]: ts=2026-09-25T15:01:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 101.669558] nix-grpc-daemon[857]: time=2026-09-25T15:01:02.212Z level=INFO msg="All 2 paths already cached" worker2 # [ 101.675378] nix-grpc-daemon[786]: ts=2026-09-25T15:01:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 101.683538] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 101.815824] nix-daemon[845]: accepted connection from pid 1165, user nix-grpc-daemon (trusted) lb # [ 101.837299] niks3-server[1035]: 2026/09/25 15:01:02 INFO Received push request method=POST path=/api/pushes worker2 # [ 101.837254] nix-daemon[845]: reaped child process 1168, status = succeeded worker2 # [ 101.839072] nix-grpc-daemon[857]: time=2026-09-25T15:01:02.381Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 101.840771] nix-grpc-daemon[857]: time=2026-09-25T15:01:02.381Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)" lb # [ 101.854769] niks3-server[1035]: 2026/09/25 15:01:02 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst worker2 # [ 101.858309] nix-grpc-daemon[857]: time=2026-09-25T15:01:02.400Z level=INFO msg="Uploading 1 narinfos" lb # [ 101.863988] niks3-server[1035]: 2026/09/25 15:01:02 INFO Received sign narinfos request method=POST path=/api/pushes/25/sign lb # [ 101.866772] niks3-server[1035]: 2026/09/25 15:01:02 INFO Signed narinfos id=25 count=1 lb # [ 101.872640] niks3-server[1035]: 2026/09/25 15:01:02 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls worker2 # [ 101.877246] nix-grpc-daemon[857]: time=2026-09-25T15:01:02.419Z level=INFO msg="Upload complete. (183ms)" lb # [ 101.881253] niks3-server[1035]: 2026/09/25 15:01:02 INFO Received complete push request method=POST path=/api/pushes/25/complete lb # [ 101.885412] niks3-server[1035]: 2026/09/25 15:01:02 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo worker2 # [ 101.884160] nix-grpc-daemon[786]: ts=2026-09-25T15:01:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_in=752 worker2 # [ 101.888640] nix-grpc-daemon[786]: ts=2026-09-25T15:01:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 101.896185] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 101.899435] nix-daemon[845]: reaped child process 1163, status = succeeded worker2 # [ 101.909364] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 103.589374] nix-daemon[845]: reaped child process 1172, status = succeeded worker2 # [ 103.652568] nix-daemon[845]: accepted connection from pid 1188, user nix-grpc-daemon (trusted) worker2 # [ 103.668461] nix-daemon[845]: reaped child process 1191, status = succeeded lb # [ 103.677909] niks3-server[1035]: 2026/09/25 15:01:04 INFO Received push request method=POST path=/api/pushes worker2 # [ 103.681719] nix-grpc-daemon[857]: time=2026-09-25T15:01:04.223Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 103.683316] nix-grpc-daemon[857]: time=2026-09-25T15:01:04.223Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)" lb # [ 103.700924] niks3-server[1035]: 2026/09/25 15:01:04 INFO Received sign narinfos request method=POST path=/api/pushes/26/sign worker2 # [ 103.702755] nix-grpc-daemon[857]: time=2026-09-25T15:01:04.245Z level=INFO msg="Uploading 1 narinfos" lb # [ 103.709395] niks3-server[1035]: 2026/09/25 15:01:04 INFO Signed narinfos id=26 count=1 lb # [ 103.714506] niks3-server[1035]: 2026/09/25 15:01:04 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls lb # [ 103.720673] niks3-server[1035]: 2026/09/25 15:01:04 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst lb # [ 103.723558] niks3-server[1035]: 2026/09/25 15:01:04 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv lb # [ 103.734779] niks3-server[1035]: 2026/09/25 15:01:04 INFO Received complete push request method=POST path=/api/pushes/26/complete worker2 # [ 103.734000] nix-grpc-daemon[857]: time=2026-09-25T15:01:04.276Z level=INFO msg="Upload complete. (216ms)" lb # [ 103.738219] niks3-server[1035]: 2026/09/25 15:01:04 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo worker2 # [ 103.740677] nix-grpc-daemon[786]: ts=2026-09-25T15:01:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=14 outputs=1 worker1 # [ 103.774467] nix-grpc-daemon[485]: ts=2026-09-25T15:01:04Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:53790 worker2 # [ 103.751624] nix-daemon[845]: reaped child process 1170, status = succeeded worker1 # [ 103.789106] nix-grpc-daemon[485]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'... worker1 # [ 103.798712] nix-grpc-daemon[485]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'... worker1 # [ 103.807118] nix-grpc-daemon[485]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'... worker1 # [ 103.827839] nix-grpc-daemon[485]: ts=2026-09-25T15:01:04Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 103.831890] nix-grpc-daemon[485]: ts=2026-09-25T15:01:04Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=0 paths=1 nar_bytes_out=232 client # [ 103.871681] nix-daemon[1100]: reaped child process 1102, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 6.08 seconds) client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.02 seconds) client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.04 seconds) client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds) (finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 6.17 seconds) subtest: an input only one worker has still reaches the builder client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only client # this derivation will be built: client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.27 seconds) client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.11 seconds) worker1: must succeed: nix-store --import < /tmp/shared/inp.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.15 seconds) worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.02 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2 client # [ 104.657864] nix-daemon[1100]: accepted connection from pid 1183, user root (trusted) client # this derivation will be built: client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker2 # [ 104.787195] nix-grpc-daemon[786]: ts=2026-09-25T15:01:05Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:50262 client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 104.818539] nix-grpc-daemon[485]: ts=2026-09-25T15:01:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 104.956776] nix-daemon[855]: accepted connection from pid 1134, user nix-grpc-daemon (trusted) lb # [ 104.950208] niks3-server[1035]: 2026/09/25 15:01:05 INFO Received push request method=POST path=/api/pushes worker1 # [ 104.976396] nix-daemon[855]: reaped child process 1137, status = succeeded worker1 # [ 104.979748] nix-grpc-daemon[905]: time=2026-09-25T15:01:05.511Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 104.981440] nix-grpc-daemon[905]: time=2026-09-25T15:01:05.511Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)" lb # [ 104.968365] niks3-server[1035]: 2026/09/25 15:01:05 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst worker1 # [ 104.998950] nix-grpc-daemon[905]: time=2026-09-25T15:01:05.530Z level=INFO msg="Uploading 1 narinfos" lb # [ 104.977078] niks3-server[1035]: 2026/09/25 15:01:05 INFO Received sign narinfos request method=POST path=/api/pushes/27/sign lb # [ 104.982255] niks3-server[1035]: 2026/09/25 15:01:05 INFO Signed narinfos id=27 count=1 lb # [ 104.988860] niks3-server[1035]: 2026/09/25 15:01:05 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls lb # [ 104.991382] niks3-server[1035]: 2026/09/25 15:01:05 INFO Received complete push request method=POST path=/api/pushes/27/complete worker1 # [ 105.019141] nix-grpc-daemon[905]: time=2026-09-25T15:01:05.551Z level=INFO msg="Upload complete. (196ms)" lb # [ 105.000623] niks3-server[1035]: 2026/09/25 15:01:05 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo worker1 # [ 105.033375] nix-grpc-daemon[485]: ts=2026-09-25T15:01:05Z level=debug event=want client=14 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=0 worker1 # [ 105.038719] nix-grpc-daemon[485]: ts=2026-09-25T15:01:05Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 105.042921] nix-grpc-daemon[485]: ts=2026-09-25T15:01:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 105.048782] nix-grpc-daemon[485]: ts=2026-09-25T15:01:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53790 client # copying 2 paths to 'lb:50051'... client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'... worker1 # [ 105.059131] nix-grpc-daemon[485]: ts=2026-09-25T15:01:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'... worker1 # [ 105.067317] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 105.182277] nix-daemon[855]: accepted connection from pid 1143, user nix-grpc-daemon (trusted) lb # [ 105.173778] niks3-server[1035]: 2026/09/25 15:01:05 INFO Received push request method=POST path=/api/pushes worker1 # [ 105.201056] nix-daemon[855]: reaped child process 1146, status = succeeded worker1 # [ 105.204257] nix-grpc-daemon[905]: time=2026-09-25T15:01:05.734Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker1 # [ 105.205780] nix-grpc-daemon[905]: time=2026-09-25T15:01:05.734Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)" worker1 # [ 105.207600] nix-grpc-daemon[905]: time=2026-09-25T15:01:05.734Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)" lb # [ 105.199659] niks3-server[1035]: 2026/09/25 15:01:05 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls lb # [ 105.205936] niks3-server[1035]: 2026/09/25 15:01:05 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst worker1 # [ 105.237765] nix-grpc-daemon[905]: time=2026-09-25T15:01:05.769Z level=INFO msg="Uploading 2 narinfos" lb # [ 105.215701] niks3-server[1035]: 2026/09/25 15:01:05 INFO Received sign narinfos request method=POST path=/api/pushes/28/sign lb # [ 105.221721] niks3-server[1035]: 2026/09/25 15:01:05 INFO Signed narinfos id=28 count=2 lb # [ 105.223469] niks3-server[1035]: 2026/09/25 15:01:05 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst lb # [ 105.235104] niks3-server[1035]: 2026/09/25 15:01:05 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls lb # [ 105.244468] niks3-server[1035]: 2026/09/25 15:01:05 INFO Received complete push request method=POST path=/api/pushes/28/complete lb # [ 105.246742] niks3-server[1035]: 2026/09/25 15:01:05 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo worker1 # [ 105.277042] nix-grpc-daemon[905]: time=2026-09-25T15:01:05.809Z level=INFO msg="Upload complete. (199ms)" lb # [ 105.253138] niks3-server[1035]: 2026/09/25 15:01:05 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo worker1 # [ 105.283821] nix-grpc-daemon[485]: ts=2026-09-25T15:01:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=0 paths=2 nar_bytes_in=984 worker1 # [ 105.294900] nix-grpc-daemon[485]: ts=2026-09-25T15:01:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 105.300241] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 105.304634] nix-daemon[855]: reaped child process 1141, status = succeeded client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051... client # uses-input-w1onlya> node-a: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker1 # [ 105.318376] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 106.368386] nix-daemon[855]: reaped child process 1150, status = succeeded worker1 # [ 106.470041] nix-daemon[855]: accepted connection from pid 1166, user nix-grpc-daemon (trusted) worker1 # [ 106.487799] nix-daemon[855]: reaped child process 1169, status = succeeded lb # [ 106.469811] niks3-server[1035]: 2026/09/25 15:01:06 INFO Received push request method=POST path=/api/pushes worker1 # [ 106.500370] nix-grpc-daemon[905]: time=2026-09-25T15:01:07.030Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 106.487671] niks3-server[1035]: 2026/09/25 15:01:07 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker1 # [ 106.518377] nix-grpc-daemon[905]: time=2026-09-25T15:01:07.050Z level=INFO msg="Uploading 1 narinfos" lb # [ 106.496300] niks3-server[1035]: 2026/09/25 15:01:07 INFO Received sign narinfos request method=POST path=/api/pushes/29/sign lb # [ 106.501779] niks3-server[1035]: 2026/09/25 15:01:07 INFO Signed narinfos id=29 count=1 lb # [ 106.508313] niks3-server[1035]: 2026/09/25 15:01:07 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls worker1 # [ 106.540053] nix-grpc-daemon[905]: time=2026-09-25T15:01:07.072Z level=INFO msg="Upload complete. (213ms)" lb # [ 106.516680] niks3-server[1035]: 2026/09/25 15:01:07 INFO Received complete push request method=POST path=/api/pushes/29/complete lb # [ 106.519139] niks3-server[1035]: 2026/09/25 15:01:07 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo worker1 # [ 106.547493] nix-grpc-daemon[485]: ts=2026-09-25T15:01:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=15 outputs=1 worker1 # [ 106.558860] nix-daemon[855]: reaped child process 1148, status = succeeded client # copying 1 paths... worker2 # [ 106.535503] nix-grpc-daemon[786]: ts=2026-09-25T15:01:07Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 106.543332] nix-grpc-daemon[786]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'... client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'... worker2 # [ 106.568440] nix-grpc-daemon[786]: ts=2026-09-25T15:01:07Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 106.572390] nix-grpc-daemon[786]: ts=2026-09-25T15:01:07Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_out=120 client # [ 106.639825] nix-daemon[1100]: reaped child process 1186, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.13 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2 client # [ 106.776818] nix-daemon[1100]: accepted connection from pid 1205, user root (trusted) client # this derivation will be built: client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker2 # [ 106.904236] nix-grpc-daemon[786]: ts=2026-09-25T15:01:07Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:50262 client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 106.934901] nix-grpc-daemon[485]: ts=2026-09-25T15:01:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 106.940495] nix-grpc-daemon[905]: time=2026-09-25T15:01:07.472Z level=INFO msg="All 1 paths already cached" worker1 # [ 106.951183] nix-grpc-daemon[485]: ts=2026-09-25T15:01:07Z level=debug event=want client=15 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=0 worker2 # [ 106.925756] nix-grpc-daemon[786]: ts=2026-09-25T15:01:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker1 # [ 106.954660] nix-grpc-daemon[485]: ts=2026-09-25T15:01:07Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 106.934227] nix-grpc-daemon[786]: ts=2026-09-25T15:01:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'... worker2 # [ 106.944837] nix-grpc-daemon[786]: ts=2026-09-25T15:01:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 106.953314] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 106.967190] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 54 ms (attempt 1/5) worker2 # [ 107.026288] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 127 ms (attempt 2/5) worker2 # [ 107.154387] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 111 ms (attempt 3/5) worker2 # [ 107.266371] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 31 ms (attempt 4/5) worker2 # [ 107.298423] nix-grpc-daemon[786]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 107.302777] nix-grpc-daemon[786]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 107.307074] nix-grpc-daemon[786]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'... worker2 # [ 107.440088] nix-daemon[845]: accepted connection from pid 1204, user nix-grpc-daemon (trusted) lb # [ 107.460729] niks3-server[1035]: 2026/09/25 15:01:07 INFO Received push request method=POST path=/api/pushes worker2 # [ 107.460080] nix-daemon[845]: reaped child process 1207, status = succeeded worker2 # [ 107.463174] nix-grpc-daemon[857]: time=2026-09-25T15:01:08.005Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 107.464878] nix-grpc-daemon[857]: time=2026-09-25T15:01:08.005Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)" worker2 # [ 107.476725] nix-grpc-daemon[857]: time=2026-09-25T15:01:08.019Z level=INFO msg="Uploading 1 narinfos" lb # [ 107.481243] niks3-server[1035]: 2026/09/25 15:01:08 INFO Received sign narinfos request method=POST path=/api/pushes/30/sign lb # [ 107.483641] niks3-server[1035]: 2026/09/25 15:01:08 INFO Signed narinfos id=30 count=1 lb # [ 107.491909] niks3-server[1035]: 2026/09/25 15:01:08 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst lb # [ 107.501238] niks3-server[1035]: 2026/09/25 15:01:08 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls worker2 # [ 107.502874] nix-grpc-daemon[857]: time=2026-09-25T15:01:08.045Z level=INFO msg="Upload complete. (188ms)" lb # [ 107.506838] niks3-server[1035]: 2026/09/25 15:01:08 INFO Received complete push request method=POST path=/api/pushes/30/complete lb # [ 107.511486] niks3-server[1035]: 2026/09/25 15:01:08 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo worker2 # [ 107.511961] nix-grpc-daemon[786]: ts=2026-09-25T15:01:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_in=568 worker2 # [ 107.515321] nix-grpc-daemon[786]: ts=2026-09-25T15:01:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 107.524682] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) worker2 # [ 107.528448] nix-daemon[845]: reaped child process 1197, status = succeeded worker2 # [ 107.540318] nix-grpc-daemon[786]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'... client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051... worker2 # [ 107.550571] nix-daemon[845]: accepted connection from pid 786, user nix-grpc-daemon (trusted) client # uses-input-w1onlyb> worker2: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 108.187021] nix-daemon[845]: reaped child process 1212, status = succeeded lb # [ 108.265948] envoy[897]: conn peer=[2001:db8:1::1]:60246 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=18 lb # [ 108.266161] envoy[897]: conn peer=[2001:db8:1::1]:60234 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3397 tx=1924 ms=1953 lb # [ 108.266226] envoy[897]: conn peer=[2001:db8:1::1]:60254 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3871 tx=2138 ms=1146 lb # [ 108.266286] envoy[897]: conn peer=[2001:db8:1::1]:55458 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=375 ms=15 lb # [ 108.266338] envoy[897]: conn peer=[2001:db8:1::1]:55470 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4335 tx=2372 ms=2184 lb # [ 108.266405] envoy[897]: conn peer=[2001:db8:1::1]:55474 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=412 tx=406 ms=15 lb # [ 108.266467] envoy[897]: conn peer=[2001:db8:1::1]:55486 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3862 tx=2065 ms=1796 lb # [ 108.266527] envoy[897]: conn peer=[2001:db8:1::1]:55498 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=16 worker2 # [ 108.293522] nix-daemon[845]: accepted connection from pid 1228, user nix-grpc-daemon (trusted) lb # [ 108.312797] niks3-server[1035]: 2026/09/25 15:01:08 INFO Received push request method=POST path=/api/pushes worker2 # [ 108.311975] nix-daemon[845]: reaped child process 1231, status = succeeded worker2 # [ 108.315423] nix-grpc-daemon[857]: time=2026-09-25T15:01:08.856Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 108.332690] niks3-server[1035]: 2026/09/25 15:01:08 INFO Received sign narinfos request method=POST path=/api/pushes/31/sign worker2 # [ 108.333134] nix-grpc-daemon[857]: time=2026-09-25T15:01:08.875Z level=INFO msg="Uploading 1 narinfos" lb # [ 108.339592] niks3-server[1035]: 2026/09/25 15:01:08 INFO Signed narinfos id=31 count=1 lb # [ 108.344362] niks3-server[1035]: 2026/09/25 15:01:08 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv lb # [ 108.352135] niks3-server[1035]: 2026/09/25 15:01:08 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls lb # [ 108.354600] niks3-server[1035]: 2026/09/25 15:01:08 INFO Received complete push request method=POST path=/api/pushes/31/complete worker2 # [ 108.356003] nix-grpc-daemon[857]: time=2026-09-25T15:01:08.898Z level=INFO msg="Upload complete. (195ms)" lb # [ 108.361576] niks3-server[1035]: 2026/09/25 15:01:08 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo client # copying 1 paths... worker2 # [ 108.364254] nix-grpc-daemon[786]: ts=2026-09-25T15:01:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=16 outputs=1 client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'... worker2 # [ 108.373991] nix-grpc-daemon[786]: ts=2026-09-25T15:01:08Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 108.377080] nix-daemon[845]: reaped child process 1209, status = succeeded worker2 # [ 108.383157] nix-grpc-daemon[786]: ts=2026-09-25T15:01:08Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 108.387613] nix-grpc-daemon[786]: ts=2026-09-25T15:01:08Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_out=120 client # [ 108.453924] nix-daemon[1100]: reaped child process 1208, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 1.81 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2 client # [ 108.587276] nix-daemon[1100]: accepted connection from pid 1227, user root (trusted) client # this derivation will be built: client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker1 # [ 108.750377] nix-grpc-daemon[485]: ts=2026-09-25T15:01:09Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:53790 client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker2 # [ 108.726196] nix-grpc-daemon[786]: ts=2026-09-25T15:01:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 108.732774] nix-grpc-daemon[857]: time=2026-09-25T15:01:09.274Z level=INFO msg="All 1 paths already cached" worker1 # [ 108.768189] nix-grpc-daemon[485]: ts=2026-09-25T15:01:09Z level=debug event=want client=16 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=0 worker1 # [ 108.773380] nix-grpc-daemon[485]: ts=2026-09-25T15:01:09Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 108.778229] nix-grpc-daemon[485]: ts=2026-09-25T15:01:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 108.782392] nix-grpc-daemon[485]: ts=2026-09-25T15:01:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 108.788617] nix-grpc-daemon[905]: time=2026-09-25T15:01:09.320Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'... worker1 # [ 108.795176] nix-grpc-daemon[485]: ts=2026-09-25T15:01:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 108.804301] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 108.927890] nix-daemon[855]: accepted connection from pid 1177, user nix-grpc-daemon (trusted) worker1 # [ 108.947405] nix-daemon[855]: reaped child process 1180, status = succeeded lb # [ 108.928362] niks3-server[1035]: 2026/09/25 15:01:09 INFO Received push request method=POST path=/api/pushes worker1 # [ 108.957743] nix-grpc-daemon[905]: time=2026-09-25T15:01:09.489Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 108.959305] nix-grpc-daemon[905]: time=2026-09-25T15:01:09.489Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)" lb # [ 108.949470] niks3-server[1035]: 2026/09/25 15:01:09 INFO Received sign narinfos request method=POST path=/api/pushes/32/sign worker1 # [ 108.977036] nix-grpc-daemon[905]: time=2026-09-25T15:01:09.508Z level=INFO msg="Uploading 1 narinfos" lb # [ 108.956080] niks3-server[1035]: 2026/09/25 15:01:09 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst lb # [ 108.959026] niks3-server[1035]: 2026/09/25 15:01:09 INFO Signed narinfos id=32 count=1 lb # [ 108.963936] niks3-server[1035]: 2026/09/25 15:01:09 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls worker1 # [ 108.995512] nix-grpc-daemon[905]: time=2026-09-25T15:01:09.527Z level=INFO msg="Upload complete. (180ms)" lb # [ 108.972508] niks3-server[1035]: 2026/09/25 15:01:09 INFO Received complete push request method=POST path=/api/pushes/32/complete lb # [ 108.974937] niks3-server[1035]: 2026/09/25 15:01:09 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo worker1 # [ 109.004304] nix-grpc-daemon[485]: ts=2026-09-25T15:01:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=0 paths=1 nar_bytes_in=568 worker1 # [ 109.012210] nix-grpc-daemon[485]: ts=2026-09-25T15:01:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 109.017613] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) worker1 # [ 109.022108] nix-daemon[855]: reaped child process 1175, status = succeeded client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051... worker1 # [ 109.031376] nix-daemon[855]: accepted connection from pid 485, user nix-grpc-daemon (trusted) client # uses-input-w1onlyc> node-a: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 110.470013] nix-daemon[855]: reaped child process 1184, status = succeeded worker1 # [ 110.578401] nix-daemon[855]: accepted connection from pid 1200, user nix-grpc-daemon (trusted) worker1 # [ 110.595264] nix-daemon[855]: reaped child process 1203, status = succeeded lb # [ 110.577812] niks3-server[1035]: 2026/09/25 15:01:11 INFO Received push request method=POST path=/api/pushes worker1 # [ 110.607475] nix-grpc-daemon[905]: time=2026-09-25T15:01:11.138Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 110.593419] niks3-server[1035]: 2026/09/25 15:01:11 INFO Received sign narinfos request method=POST path=/api/pushes/33/sign worker1 # [ 110.622089] nix-grpc-daemon[905]: time=2026-09-25T15:01:11.153Z level=INFO msg="Uploading 1 narinfos" lb # [ 110.600200] niks3-server[1035]: 2026/09/25 15:01:11 INFO Signed narinfos id=33 count=1 lb # [ 110.605488] niks3-server[1035]: 2026/09/25 15:01:11 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv lb # [ 110.615924] niks3-server[1035]: 2026/09/25 15:01:11 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls worker1 # [ 110.644406] nix-grpc-daemon[905]: time=2026-09-25T15:01:11.176Z level=INFO msg="Upload complete. (203ms)" lb # [ 110.621231] niks3-server[1035]: 2026/09/25 15:01:11 INFO Received complete push request method=POST path=/api/pushes/33/complete lb # [ 110.625706] niks3-server[1035]: 2026/09/25 15:01:11 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo worker1 # [ 110.652570] nix-grpc-daemon[485]: ts=2026-09-25T15:01:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53790 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=17 outputs=1 client # copying 1 paths... worker1 # [ 110.663897] nix-grpc-daemon[485]: ts=2026-09-25T15:01:11Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:53790 worker1 # [ 110.667610] nix-daemon[855]: reaped child process 1182, status = succeeded client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'... worker2 # [ 110.656156] nix-grpc-daemon[786]: ts=2026-09-25T15:01:11Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 worker2 # [ 110.663349] nix-grpc-daemon[786]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'... worker2 # [ 110.672817] nix-grpc-daemon[786]: ts=2026-09-25T15:01:11Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:50262 duration_s=0 paths=1 nar_bytes_out=120 client # [ 110.740924] nix-daemon[1100]: reaped child process 1230, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.29 seconds) (finished: subtest: an input only one worker has still reaches the builder, in 6.78 seconds) subtest: upload whose reference the worker lacks is completed from the cache client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache client # this derivation will be built: client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.29 seconds) client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache client # this derivation will be built: client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.27 seconds) client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.11 seconds) worker1: must succeed: nix-store --import < /tmp/shared/ref.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.15 seconds) worker2 # [ 111.593988] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 111.969039] nix-grpc-daemon[786]: ts=2026-09-25T15:01:12Z level=info event=signal_exit worker2 # [ 113.950268] nix-grpc-daemon[786]: ts=2026-09-25T15:01:14Z level=info event=scheduler_restarting worker1 # [ 113.986598] nix-grpc-daemon[485]: ts=2026-09-25T15:01:14Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:60128 worker1 # [ 113.991300] nix-grpc-daemon[485]: ts=2026-09-25T15:01:14Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 113.973345] nix-grpc-daemon[786]: ts=2026-09-25T15:01:14Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 113.987129] nix-daemon[845]: reaped child process 847, status = succeeded worker2 # [ 115.665782] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 115.668321] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 115.672662] systemd[1]: nix-grpc-daemon.service: Consumed 2.891s CPU time over 1min 21.885s wall clock time, 65.2M memory peak, 191.5K incoming IP traffic, 286.2K outgoing IP traffic. lb: waiting until exactly ['worker1'] are healthy members of aarch64-linux lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: (finished: waiting until exactly ['worker1'] are healthy members of aarch64-linux, in 0.06 seconds) client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2 worker1 # [ 115.944553] nix-grpc-daemon[485]: ts=2026-09-25T15:01:16Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53790 client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache worker1 # [ 115.948962] nix-grpc-daemon[485]: ts=2026-09-25T15:01:16Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:53790 client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.20 seconds) worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.02 seconds) worker2 # [ 115.993816] systemd[1]: Starting scheduler-token.service... worker2 # [ 116.047655] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 116.052399] systemd[1]: Finished scheduler-token.service. worker2 # [ 116.053591] systemd[1]: scheduler-token.service: Consumed 32ms CPU time over 56ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 116.059611] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 116.172509] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:16Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 116.174958] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:16Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 116.179164] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:16Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 116.184064] nix-grpc-daemon[1256]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 116.185663] nix-grpc-daemon[1256]: E0000 00:00:1790348476.726265 1256 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 116.190640] nix-grpc-daemon[1256]: E0000 00:00:1790348476.732898 1256 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 116.235633] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:16Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 116.238145] nix-grpc-daemon[1256]: E0000 00:00:1790348476.742291 1296 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 116.243662] nix-grpc-daemon[1256]: E0000 00:00:1790348476.760099 1296 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 116.248165] nix-grpc-daemon[1256]: E0000 00:00:1790348476.760260 1296 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 116.277398] nix-grpc-daemon[485]: ts=2026-09-25T15:01:16Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:60128 worker2 # [ 116.251465] nix-grpc-daemon[1256]: E0000 00:00:1790348476.777125 1296 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 116.254233] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 116.255095] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 116.282785] nix-grpc-daemon[485]: ts=2026-09-25T15:01:16Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 116.321344] systemd[1]: Stopping Nix worker-protocol over gRPC... worker1 # [ 116.851441] nix-grpc-daemon[485]: ts=2026-09-25T15:01:17Z level=info event=signal_exit lb # [ 117.520938] niks3-server[1035]: 2026/09/25 15:01:18 INFO lead: released remote=192.168.1.3:45362 lb # [ 118.267003] envoy[897]: conn peer=[2001:db8:1::1]:55508 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3723 tx=2105 ms=1493 lb # [ 118.269914] envoy[897]: conn peer=[2001:db8:1::1]:55520 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=14 lb # [ 118.272420] envoy[897]: conn peer=[2001:db8:1::1]:55536 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=329 ms=29 lb # [ 118.274863] envoy[897]: conn peer=[2001:db8:1::1]:55522 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3724 tx=2103 ms=1961 lb # [ 118.277493] envoy[897]: conn peer=[2001:db8:1::4]:32886 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=34292 tx=2498 ms=79968 lb # [ 118.280134] envoy[897]: conn peer=[2001:db8:1::1]:58536 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=15 worker1 # [ 118.543760] nix-grpc-daemon[485]: ts=2026-09-25T15:01:19Z level=info event=scheduler_restarting worker2 # [ 118.518195] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:19Z level=info event=scheduler_restarting addr=lb:50051 lb # [ 118.519931] envoy[897]: [2026-09-25 15:01:19.042][897][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream} worker1 # [ 118.548062] nix-grpc-daemon[485]: ts=2026-09-25T15:01:19Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 118.554551] nix-grpc-daemon[485]: ts=2026-09-25T15:01:19Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:60128 worker1 # [ 118.557927] nix-grpc-daemon[485]: ts=2026-09-25T15:01:19Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 118.560436] nix-grpc-daemon[485]: ts=2026-09-25T15:01:19Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:60128 worker1 # [ 118.562727] nix-grpc-daemon[485]: ts=2026-09-25T15:01:19Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 118.582659] nix-daemon[855]: reaped child process 857, status = succeeded worker2 # [ 118.718652] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 118.721997] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 118.763872] nix-grpc-daemon[485]: ts=2026-09-25T15:01:19Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 118.922797] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 118.926027] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 119.126817] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 119.129993] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 119.331241] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 119.334758] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 119.536723] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 119.539012] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 119.662345] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 119.666844] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 119.670195] systemd[1]: nix-grpc-daemon.service: Consumed 2.408s CPU time over 1min 49.147s wall clock time, 66.2M memory peak, 425K incoming IP traffic, 262.3K outgoing IP traffic. lb: waiting until exactly ['worker2'] are healthy members of aarch64-linux lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: (finished: waiting until exactly ['worker2'] are healthy members of aarch64-linux, in 0.04 seconds) client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2 worker2 # [ 119.737379] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 119.740415] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 119.864606] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:38944 worker2 # [ 119.873326] nix-daemon[845]: accepted connection from pid 1256, user nix-grpc-daemon (trusted) worker2 # [ 119.886737] nix-grpc-daemon[1256]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 0 ms (attempt 1/5) worker2 # [ 119.892086] nix-grpc-daemon[1256]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 101 ms (attempt 2/5) worker2 # [ 119.941130] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 119.943951] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 119.994145] nix-grpc-daemon[1256]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 369 ms (attempt 3/5) worker2 # [ 120.144793] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 120.147618] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 120.348557] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 120.351288] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 120.364394] nix-grpc-daemon[1256]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 259 ms (attempt 4/5) worker2 # [ 120.553013] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 120.555269] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 120.629022] nix-grpc-daemon[1256]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 120.632875] nix-grpc-daemon[1256]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org client # copying 2 paths... client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'... worker2 # [ 120.646971] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38944 client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'... worker2 # [ 120.656135] nix-daemon[845]: accepted connection from pid 1256, user nix-grpc-daemon (trusted) worker2 # [ 120.673338] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=info event=niks3_push_started pid=1316 worker2 # [ 120.756624] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 120.780192] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 120.808735] nix-grpc-daemon[1316]: time=2026-09-25T15:01:21.349Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 120.917786] nix-daemon[845]: accepted connection from pid 1329, user nix-grpc-daemon (trusted) lb # [ 120.940785] niks3-server[1035]: 2026/09/25 15:01:21 INFO Received push request method=POST path=/api/pushes worker2 # [ 120.939535] nix-daemon[845]: reaped child process 1332, status = succeeded worker2 # [ 120.944110] nix-grpc-daemon[1316]: time=2026-09-25T15:01:21.485Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker2 # [ 120.945685] nix-grpc-daemon[1316]: time=2026-09-25T15:01:21.485Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)" worker2 # [ 120.947510] nix-grpc-daemon[1316]: time=2026-09-25T15:01:21.485Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)" worker2 # [ 120.960824] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 120.963001] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 120.985423] niks3-server[1035]: 2026/09/25 15:01:21 INFO Received sign narinfos request method=POST path=/api/pushes/34/sign worker2 # [ 120.987639] nix-grpc-daemon[1316]: time=2026-09-25T15:01:21.528Z level=INFO msg="Uploading 2 narinfos" lb # [ 120.993389] niks3-server[1035]: 2026/09/25 15:01:21 INFO Signed narinfos id=34 count=2 lb # [ 121.000047] niks3-server[1035]: 2026/09/25 15:01:21 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst lb # [ 121.009771] niks3-server[1035]: 2026/09/25 15:01:21 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls lb # [ 121.017950] niks3-server[1035]: 2026/09/25 15:01:21 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls lb # [ 121.031880] niks3-server[1035]: 2026/09/25 15:01:21 INFO Received complete push request method=POST path=/api/pushes/34/complete lb # [ 121.041358] niks3-server[1035]: 2026/09/25 15:01:21 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo lb # [ 121.050114] niks3-server[1035]: 2026/09/25 15:01:21 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst worker2 # [ 121.049535] nix-grpc-daemon[1316]: time=2026-09-25T15:01:21.592Z level=INFO msg="Upload complete. (241ms)" lb # [ 121.055603] niks3-server[1035]: 2026/09/25 15:01:21 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo worker2 # [ 121.060256] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38944 duration_s=0 paths=2 nar_bytes_in=312 worker2 # [ 121.069088] nix-daemon[845]: reaped child process 1314, status = succeeded client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.38 seconds) worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.02 seconds) worker1 # [ 121.168836] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 121.163968] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 121.166856] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 121.203096] niks3-server[1035]: 2026/09/25 15:01:21 INFO lead: acquired remote=192.168.1.4:36834 worker2 # [ 121.201825] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=info event=scheduler_take_over worker1 # [ 121.285035] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:21Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 121.287302] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:21Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 121.291200] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:21Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 121.296072] nix-grpc-daemon[1226]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 121.297578] nix-grpc-daemon[1226]: E0000 00:00:1790348481.827776 1226 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 121.302331] nix-grpc-daemon[1226]: E0000 00:00:1790348481.834198 1226 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 121.347293] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:21Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 121.349773] nix-grpc-daemon[1226]: E0000 00:00:1790348481.843832 1267 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 121.352855] nix-grpc-daemon[1226]: E0000 00:00:1790348481.861525 1267 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 121.358791] nix-grpc-daemon[1226]: E0000 00:00:1790348481.861688 1267 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 121.363960] nix-grpc-daemon[1226]: E0000 00:00:1790348481.878349 1267 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 121.366902] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 121.367891] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:21Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 121.369325] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2 # [ 121.368039] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 121.371183] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 121.567635] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 121.551092] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:22Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:48022 worker2 # [ 121.555126] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:22Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker2 # [ 121.572053] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 121.580120] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:22Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 121.582949] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:22Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:48022 worker2 # [ 121.586027] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:22Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: (finished: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux, in 1.09 seconds) (finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.66 seconds) subtest: a niks3 restart does not cost the leader its role lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb # [ 122.503111] systemd[1]: Stopping niks3 server... worker2 # [ 122.499157] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:23Z level=info event=scheduler_lock_lost status=200 worker1 # [ 122.526813] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:23Z level=info event=scheduler_lock_lost status=200 lb # [ 122.503798] niks3-server[1035]: 2026/09/25 15:01:23 INFO Shutdown signal received, draining in-flight requests timeout=10s lb # [ 122.503902] niks3-server[1035]: 2026/09/25 15:01:23 INFO lead: released remote=192.168.1.4:36834 lb # [ 122.528484] systemd[1]: niks3.service: Deactivated successfully. lb # [ 122.530939] systemd[1]: Stopped niks3 server. lb # [ 122.531887] systemd[1]: niks3.service: Consumed 424ms CPU time over 1min 31.280s wall clock time, 28M memory peak, 93.7K incoming IP traffic, 135.5K outgoing IP traffic. lb # [ 122.541482] systemd[1]: Starting niks3 server... lb # [ 122.599124] niks3-server[1213]: 2026/09/25 15:01:23 goose: no migrations to run. current version: 20260923120000 lb # [ 122.605373] niks3-server[1213]: 2026/09/25 15:01:23 OK 1_commit_pending_closure.sql (4.35ms) lb # [ 122.608555] niks3-server[1213]: 2026/09/25 15:01:23 OK 2_object_stats_trigger.sql (1.81ms) lb # [ 122.611570] niks3-server[1213]: 2026/09/25 15:01:23 OK 3_commit_push.sql (2.68ms) lb # [ 122.613907] niks3-server[1213]: 2026/09/25 15:01:23 goose: up to current file version: 3 lb # [ 122.615054] niks3-server[1213]: 2026/09/25 15:01:23 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key lb # [ 122.625269] niks3-server[1213]: 2026/09/25 15:01:23 INFO Read proxy enabled — serving cache objects from S3 lb # [ 122.626649] niks3-server[1213]: 2026/09/25 15:01:23 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 122.627966] niks3-server[1213]: 2026/09/25 15:01:23 INFO systemd watchdog enabled interval=15s lb # [ 122.633863] niks3-server[1213]: 2026/09/25 15:01:23 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 122.635325] systemd[1]: Started niks3 server. lb: waiting for unit niks3.service lb: (finished: waiting for unit niks3.service, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.04 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2 worker1 # [ 123.014740] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 123.024375] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) client # don't know how to build these paths: client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker2 # [ 123.011660] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:38944 worker2 # [ 123.031448] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:23Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=0 worker1 # [ 123.061225] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker2 # [ 123.035287] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:23Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 123.070173] nix-grpc-daemon[1226]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 79 ms (attempt 1/5) worker1 # [ 123.154180] nix-grpc-daemon[1226]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 16 ms (attempt 2/5) worker1 # [ 123.171175] nix-grpc-daemon[1226]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 27 ms (attempt 3/5) worker1 # [ 123.198941] nix-grpc-daemon[1226]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 570 ms (attempt 4/5) lb # [ 123.506515] niks3-server[1213]: 2026/09/25 15:01:24 INFO lead: acquired remote=192.168.1.4:36680 worker1 # [ 123.771019] nix-grpc-daemon[1226]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 123.774763] nix-grpc-daemon[1226]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 123.785850] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:33728 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'... worker1 # [ 123.794570] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 123.803103] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 123.818648] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:24Z level=info event=niks3_push_started pid=1288 worker1 # [ 123.944873] nix-grpc-daemon[1288]: time=2026-09-25T15:01:24.476Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 124.056483] nix-daemon[855]: accepted connection from pid 1300, user nix-grpc-daemon (trusted) worker1 # [ 124.076360] nix-daemon[855]: reaped child process 1303, status = succeeded lb # [ 124.053978] niks3-server[1213]: 2026/09/25 15:01:24 INFO Received push request method=POST path=/api/pushes worker1 # [ 124.087519] nix-grpc-daemon[1288]: time=2026-09-25T15:01:24.618Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 124.091406] nix-grpc-daemon[1288]: time=2026-09-25T15:01:24.618Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)" lb # [ 124.090544] niks3-server[1213]: 2026/09/25 15:01:24 INFO Received sign narinfos request method=POST path=/api/pushes/35/sign lb # [ 124.114394] niks3-server[1213]: 2026/09/25 15:01:24 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls lb # [ 124.124125] niks3-server[1213]: 2026/09/25 15:01:24 INFO Signed narinfos id=35 count=1 worker1 # [ 124.151396] nix-grpc-daemon[1288]: time=2026-09-25T15:01:24.682Z level=INFO msg="Uploading 1 narinfos" lb # [ 124.126503] niks3-server[1213]: 2026/09/25 15:01:24 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst lb # [ 124.140116] niks3-server[1213]: 2026/09/25 15:01:24 INFO Received complete push request method=POST path=/api/pushes/35/complete worker1 # [ 124.172108] nix-grpc-daemon[1288]: time=2026-09-25T15:01:24.704Z level=INFO msg="Upload complete. (223ms)" lb # [ 124.152842] niks3-server[1213]: 2026/09/25 15:01:24 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo worker1 # [ 124.180441] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=0 paths=1 nar_bytes_in=544 worker1 # [ 124.183753] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 124.194616] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 124.198743] nix-daemon[855]: reaped child process 1286, status = succeeded client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051... client # farm-a> node-a: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker1 # [ 124.208352] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-after-niks3-restart worker1 # [ 125.034792] nix-daemon[855]: reaped child process 1307, status = succeeded worker1 # [ 125.150594] nix-daemon[855]: accepted connection from pid 1323, user nix-grpc-daemon (trusted) worker1 # [ 125.166575] nix-daemon[855]: reaped child process 1326, status = succeeded lb # [ 125.149169] niks3-server[1213]: 2026/09/25 15:01:25 INFO Received push request method=POST path=/api/pushes worker1 # [ 125.180385] nix-grpc-daemon[1288]: time=2026-09-25T15:01:25.711Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 125.181959] nix-grpc-daemon[1288]: time=2026-09-25T15:01:25.711Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)" lb # [ 125.178703] niks3-server[1213]: 2026/09/25 15:01:25 INFO Received sign narinfos request method=POST path=/api/pushes/36/sign worker1 # [ 125.208864] nix-grpc-daemon[1288]: time=2026-09-25T15:01:25.740Z level=INFO msg="Uploading 1 narinfos" lb # [ 125.188842] niks3-server[1213]: 2026/09/25 15:01:25 INFO Signed narinfos id=36 count=1 lb # [ 125.202670] niks3-server[1213]: 2026/09/25 15:01:25 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls lb # [ 125.208433] niks3-server[1213]: 2026/09/25 15:01:25 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv lb # [ 125.219575] niks3-server[1213]: 2026/09/25 15:01:25 INFO Received complete push request method=POST path=/api/pushes/36/complete lb # [ 125.225068] niks3-server[1213]: 2026/09/25 15:01:25 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst lb # [ 125.229192] niks3-server[1213]: 2026/09/25 15:01:25 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo worker1 # [ 125.255603] nix-grpc-daemon[1288]: time=2026-09-25T15:01:25.787Z level=INFO msg="Upload complete. (237ms)" worker1 # [ 125.264614] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=1 worker2 # [ 125.239975] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:25Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=0 worker2 # [ 125.243971] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:25Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 125.274444] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 125.278523] nix-daemon[855]: reaped child process 1305, status = succeeded worker1 # [ 125.283706] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 125.291606] nix-grpc-daemon[1288]: time=2026-09-25T15:01:25.823Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'... worker1 # [ 125.297646] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 125.305052] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 125.420744] nix-daemon[855]: accepted connection from pid 1330, user nix-grpc-daemon (trusted) worker1 # [ 125.436457] nix-daemon[855]: reaped child process 1333, status = succeeded lb # [ 125.418644] niks3-server[1213]: 2026/09/25 15:01:25 INFO Received push request method=POST path=/api/pushes worker1 # [ 125.449781] nix-grpc-daemon[1288]: time=2026-09-25T15:01:25.981Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 125.451355] nix-grpc-daemon[1288]: time=2026-09-25T15:01:25.981Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)" lb # [ 125.438742] niks3-server[1213]: 2026/09/25 15:01:25 INFO Received sign narinfos request method=POST path=/api/pushes/37/sign worker1 # [ 125.467722] nix-grpc-daemon[1288]: time=2026-09-25T15:01:25.999Z level=INFO msg="Uploading 1 narinfos" lb # [ 125.446591] niks3-server[1213]: 2026/09/25 15:01:25 INFO Signed narinfos id=37 count=1 lb # [ 125.452101] niks3-server[1213]: 2026/09/25 15:01:25 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst lb # [ 125.465147] niks3-server[1213]: 2026/09/25 15:01:25 INFO Received complete push request method=POST path=/api/pushes/37/complete lb # [ 125.469090] niks3-server[1213]: 2026/09/25 15:01:25 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls worker1 # [ 125.497577] nix-grpc-daemon[1288]: time=2026-09-25T15:01:26.029Z level=INFO msg="Upload complete. (183ms)" lb # [ 125.475327] niks3-server[1213]: 2026/09/25 15:01:25 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo worker1 # [ 125.504281] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=0 paths=1 nar_bytes_in=656 worker1 # [ 125.507534] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 125.518037] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 125.522145] nix-daemon[855]: reaped child process 1328, status = succeeded client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051... client # farm-b> node-a: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker1 # [ 125.532290] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-after-niks3-restart worker1 # [ 126.390351] nix-daemon[855]: reaped child process 1337, status = succeeded worker1 # [ 126.446942] nix-daemon[855]: accepted connection from pid 1353, user nix-grpc-daemon (trusted) worker1 # [ 126.463217] nix-daemon[855]: reaped child process 1356, status = succeeded lb # [ 126.446091] niks3-server[1213]: 2026/09/25 15:01:26 INFO Received push request method=POST path=/api/pushes worker1 # [ 126.476151] nix-grpc-daemon[1288]: time=2026-09-25T15:01:27.007Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 126.477685] nix-grpc-daemon[1288]: time=2026-09-25T15:01:27.007Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)" lb # [ 126.472932] niks3-server[1213]: 2026/09/25 15:01:26 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker1 # [ 126.502363] nix-grpc-daemon[1288]: time=2026-09-25T15:01:27.034Z level=INFO msg="Uploading 1 narinfos" lb # [ 126.480205] niks3-server[1213]: 2026/09/25 15:01:26 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst lb # [ 126.488269] niks3-server[1213]: 2026/09/25 15:01:27 INFO Received sign narinfos request method=POST path=/api/pushes/38/sign lb # [ 126.496539] niks3-server[1213]: 2026/09/25 15:01:27 INFO Signed narinfos id=38 count=1 lb # [ 126.498332] niks3-server[1213]: 2026/09/25 15:01:27 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls worker1 # [ 126.526638] nix-grpc-daemon[1288]: time=2026-09-25T15:01:27.058Z level=INFO msg="Upload complete. (186ms)" lb # [ 126.503827] niks3-server[1213]: 2026/09/25 15:01:27 INFO Received complete push request method=POST path=/api/pushes/38/complete lb # [ 126.508839] niks3-server[1213]: 2026/09/25 15:01:27 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo worker1 # [ 126.533879] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=1 worker2 # [ 126.511232] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:27Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=0 worker2 # [ 126.514779] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:27Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 126.545542] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 126.549496] nix-daemon[855]: reaped child process 1335, status = succeeded worker1 # [ 126.555384] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 126.563274] nix-grpc-daemon[1288]: time=2026-09-25T15:01:27.095Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'... worker1 # [ 126.569300] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 126.576955] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 126.690482] nix-daemon[855]: accepted connection from pid 1360, user nix-grpc-daemon (trusted) lb # [ 126.683741] niks3-server[1213]: 2026/09/25 15:01:27 INFO Received push request method=POST path=/api/pushes worker1 # [ 126.709155] nix-daemon[855]: reaped child process 1363, status = succeeded worker1 # [ 126.712688] nix-grpc-daemon[1288]: time=2026-09-25T15:01:27.244Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 126.714175] nix-grpc-daemon[1288]: time=2026-09-25T15:01:27.244Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)" lb # [ 126.702338] niks3-server[1213]: 2026/09/25 15:01:27 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls worker1 # [ 126.731167] nix-grpc-daemon[1288]: time=2026-09-25T15:01:27.262Z level=INFO msg="Uploading 1 narinfos" lb # [ 126.709150] niks3-server[1213]: 2026/09/25 15:01:27 INFO Received sign narinfos request method=POST path=/api/pushes/39/sign lb # [ 126.711470] niks3-server[1213]: 2026/09/25 15:01:27 INFO Signed narinfos id=39 count=1 lb # [ 126.720573] niks3-server[1213]: 2026/09/25 15:01:27 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst worker1 # [ 126.752793] nix-grpc-daemon[1288]: time=2026-09-25T15:01:27.284Z level=INFO msg="Upload complete. (167ms)" lb # [ 126.729803] niks3-server[1213]: 2026/09/25 15:01:27 INFO Received complete push request method=POST path=/api/pushes/39/complete lb # [ 126.733564] niks3-server[1213]: 2026/09/25 15:01:27 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo worker1 # [ 126.760201] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=0 paths=1 nar_bytes_in=784 worker1 # [ 126.768207] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 126.774601] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 126.776568] nix-daemon[855]: reaped child process 1358, status = succeeded client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051... client # farm-top> node-a: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker1 # [ 126.786998] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-after-niks3-restart worker1 # [ 127.621375] nix-daemon[855]: reaped child process 1367, status = succeeded worker1 # [ 127.726711] nix-daemon[855]: accepted connection from pid 1383, user nix-grpc-daemon (trusted) worker1 # [ 127.746106] nix-daemon[855]: reaped child process 1386, status = succeeded lb # [ 127.725688] niks3-server[1213]: 2026/09/25 15:01:28 INFO Received push request method=POST path=/api/pushes worker1 # [ 127.756414] nix-grpc-daemon[1288]: time=2026-09-25T15:01:28.287Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 127.758114] nix-grpc-daemon[1288]: time=2026-09-25T15:01:28.287Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)" lb # [ 127.755851] niks3-server[1213]: 2026/09/25 15:01:28 INFO Received sign narinfos request method=POST path=/api/pushes/40/sign worker1 # [ 127.782962] nix-grpc-daemon[1288]: time=2026-09-25T15:01:28.314Z level=INFO msg="Uploading 1 narinfos" lb # [ 127.766089] niks3-server[1213]: 2026/09/25 15:01:28 INFO Signed narinfos id=40 count=1 lb # [ 127.767856] niks3-server[1213]: 2026/09/25 15:01:28 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv lb # [ 127.776525] niks3-server[1213]: 2026/09/25 15:01:28 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst lb # [ 127.779433] niks3-server[1213]: 2026/09/25 15:01:28 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls lb # [ 127.789055] niks3-server[1213]: 2026/09/25 15:01:28 INFO Received complete push request method=POST path=/api/pushes/40/complete worker1 # [ 127.815852] nix-grpc-daemon[1288]: time=2026-09-25T15:01:28.347Z level=INFO msg="Upload complete. (221ms)" lb # [ 127.792923] niks3-server[1213]: 2026/09/25 15:01:28 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo worker1 # [ 127.824615] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=1 worker1 # [ 127.833485] nix-daemon[855]: reaped child process 1365, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2, in 5.01 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath, in 0.14 seconds) (finished: subtest: a niks3 restart does not cost the leader its role, in 5.56 seconds) subtest: scheduler down: the other node takes the lock and keeps it lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2 # [ 128.134076] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 128.199436] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:28Z level=info event=signal_exit lb # [ 128.267318] envoy[897]: conn peer=[2001:db8:1::3]:51224 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2359 tx=1186 ms=80941 lb # [ 128.270194] envoy[897]: conn peer=[2001:db8:1::1]:58548 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1005 tx=305 ms=1204 lb # [ 128.272789] envoy[897]: conn peer=[2001:db8:1::1]:53302 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9680 tx=4400 ms=4823 lb # [ 128.512140] niks3-server[1213]: 2026/09/25 15:01:29 INFO lead: released remote=192.168.1.4:36680 worker1 # [ 129.533429] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:30Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 129.509814] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:30Z level=info event=scheduler_restarting worker2 # [ 129.512769] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:30Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 129.515014] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:30Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:48022 worker2 # [ 129.517939] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:30Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 129.520173] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:30Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:48022 worker2 # [ 129.522634] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:30Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 129.538141] nix-daemon[845]: reaped child process 1308, status = succeeded worker1 # [ 129.733569] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 129.736695] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 129.717614] nix-grpc-daemon[1256]: ts=2026-09-25T15:01:30Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker1 # [ 129.937064] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 129.939872] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 130.140416] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 130.143232] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 130.343843] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 130.346760] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 130.548222] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 130.550341] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 130.748639] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 130.751453] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 130.746167] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 130.750479] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 130.751962] systemd[1]: nix-grpc-daemon.service: Consumed 495ms CPU time over 14.687s wall clock time, 33.5M memory peak, 45.8K incoming IP traffic, 70.5K outgoing IP traffic. worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker1 # [ 130.952198] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 130.954951] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 131.155498] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 131.158371] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 131.358860] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 131.361694] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 131.562951] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 131.565209] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 131.763166] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 131.766007] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker1 # [ 131.966426] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 131.969231] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 132.169624] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 132.172348] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 132.372756] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 132.375462] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 132.576537] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 132.579091] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 132.779555] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 132.782308] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker1 # [ 132.982782] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 132.985564] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 133.186005] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 133.188722] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 133.389197] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 133.391916] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 133.530162] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:34Z level=info event=scheduler_take_over lb # [ 133.507179] niks3-server[1213]: 2026/09/25 15:01:34 INFO lead: acquired remote=192.168.1.3:37636 worker1 # [ 133.593272] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 133.595427] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 133.793652] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 133.796386] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker1 # [ 133.996871] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 133.999617] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 134.200056] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 134.202718] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 134.403257] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 134.406028] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 134.607203] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:35Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 134.617685] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:35Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:38230 worker1 # [ 134.620521] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:35Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2 client # don't know how to build these paths: client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker1 # [ 135.194212] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 135.199727] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 135.211082] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:35Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=0 worker1 # [ 135.216259] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:35Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 135.220685] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 135.224626] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:33728 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'... worker1 # [ 135.231377] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 135.240143] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 135.361646] nix-daemon[855]: accepted connection from pid 1413, user nix-grpc-daemon (trusted) worker1 # [ 135.377825] nix-daemon[855]: reaped child process 1416, status = succeeded lb # [ 135.359958] niks3-server[1213]: 2026/09/25 15:01:35 INFO Received push request method=POST path=/api/pushes worker1 # [ 135.390627] nix-grpc-daemon[1288]: time=2026-09-25T15:01:35.921Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 135.392318] nix-grpc-daemon[1288]: time=2026-09-25T15:01:35.922Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)" lb # [ 135.378797] niks3-server[1213]: 2026/09/25 15:01:35 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst worker1 # [ 135.408774] nix-grpc-daemon[1288]: time=2026-09-25T15:01:35.940Z level=INFO msg="Uploading 1 narinfos" lb # [ 135.387350] niks3-server[1213]: 2026/09/25 15:01:35 INFO Received sign narinfos request method=POST path=/api/pushes/41/sign lb # [ 135.392521] niks3-server[1213]: 2026/09/25 15:01:35 INFO Signed narinfos id=41 count=1 lb # [ 135.400955] niks3-server[1213]: 2026/09/25 15:01:35 INFO Received complete push request method=POST path=/api/pushes/41/complete lb # [ 135.403374] niks3-server[1213]: 2026/09/25 15:01:35 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls lb # [ 135.414777] niks3-server[1213]: 2026/09/25 15:01:35 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo worker1 # [ 135.454230] nix-grpc-daemon[1288]: time=2026-09-25T15:01:35.986Z level=INFO msg="Upload complete. (205ms)" worker1 # [ 135.460269] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=0 paths=1 nar_bytes_in=520 worker1 # [ 135.463582] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 135.475100] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 135.479406] nix-daemon[855]: reaped child process 1411, status = succeeded client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051... client # farm-a> node-a: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker1 # [ 135.488597] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-on-standby worker1 # [ 136.366485] nix-daemon[855]: reaped child process 1420, status = succeeded worker1 # [ 136.428048] nix-daemon[855]: accepted connection from pid 1436, user nix-grpc-daemon (trusted) worker1 # [ 136.450373] nix-daemon[855]: reaped child process 1439, status = succeeded lb # [ 136.426470] niks3-server[1213]: 2026/09/25 15:01:36 INFO Received push request method=POST path=/api/pushes worker1 # [ 136.459982] nix-grpc-daemon[1288]: time=2026-09-25T15:01:36.989Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 136.461719] nix-grpc-daemon[1288]: time=2026-09-25T15:01:36.989Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)" lb # [ 136.457977] niks3-server[1213]: 2026/09/25 15:01:36 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst worker1 # [ 136.489816] nix-grpc-daemon[1288]: time=2026-09-25T15:01:37.021Z level=INFO msg="Uploading 1 narinfos" lb # [ 136.467792] niks3-server[1213]: 2026/09/25 15:01:36 INFO Received sign narinfos request method=POST path=/api/pushes/42/sign lb # [ 136.475465] niks3-server[1213]: 2026/09/25 15:01:36 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv lb # [ 136.482352] niks3-server[1213]: 2026/09/25 15:01:36 INFO Signed narinfos id=42 count=1 lb # [ 136.489685] niks3-server[1213]: 2026/09/25 15:01:36 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls worker1 # [ 136.520289] nix-grpc-daemon[1288]: time=2026-09-25T15:01:37.052Z level=INFO msg="Upload complete. (222ms)" lb # [ 136.497000] niks3-server[1213]: 2026/09/25 15:01:37 INFO Received complete push request method=POST path=/api/pushes/42/complete lb # [ 136.500742] niks3-server[1213]: 2026/09/25 15:01:37 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo worker1 # [ 136.535135] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=1 worker1 # [ 136.551125] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:37Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=0 worker1 # [ 136.554426] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:37Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 136.560132] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 136.564229] nix-daemon[855]: reaped child process 1418, status = succeeded worker1 # [ 136.566441] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 136.574720] nix-grpc-daemon[1288]: time=2026-09-25T15:01:37.106Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'... worker1 # [ 136.581398] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 136.589578] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 136.707236] nix-daemon[855]: accepted connection from pid 1445, user nix-grpc-daemon (trusted) worker1 # [ 136.723263] nix-daemon[855]: reaped child process 1448, status = succeeded lb # [ 136.705764] niks3-server[1213]: 2026/09/25 15:01:37 INFO Received push request method=POST path=/api/pushes worker1 # [ 136.735303] nix-grpc-daemon[1288]: time=2026-09-25T15:01:37.266Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 136.737000] nix-grpc-daemon[1288]: time=2026-09-25T15:01:37.266Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)" worker1 # [ 136.752849] nix-grpc-daemon[1288]: time=2026-09-25T15:01:37.284Z level=INFO msg="Uploading 1 narinfos" lb # [ 136.725566] niks3-server[1213]: 2026/09/25 15:01:37 INFO Received sign narinfos request method=POST path=/api/pushes/43/sign lb # [ 136.733654] niks3-server[1213]: 2026/09/25 15:01:37 INFO Signed narinfos id=43 count=1 lb # [ 136.735341] niks3-server[1213]: 2026/09/25 15:01:37 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls lb # [ 136.746624] niks3-server[1213]: 2026/09/25 15:01:37 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst worker1 # [ 136.774231] nix-grpc-daemon[1288]: time=2026-09-25T15:01:37.306Z level=INFO msg="Upload complete. (175ms)" lb # [ 136.751600] niks3-server[1213]: 2026/09/25 15:01:37 INFO Received complete push request method=POST path=/api/pushes/43/complete worker1 # [ 136.780181] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=0 paths=1 nar_bytes_in=640 lb # [ 136.757646] niks3-server[1213]: 2026/09/25 15:01:37 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo worker1 # [ 136.790203] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 136.794705] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 136.798434] nix-daemon[855]: reaped child process 1443, status = succeeded client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051... worker1 # [ 136.807295] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) client # farm-b> node-a: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-on-standby worker1 # [ 138.047417] nix-daemon[855]: reaped child process 1452, status = succeeded worker1 # [ 138.147833] nix-daemon[855]: accepted connection from pid 1468, user nix-grpc-daemon (trusted) worker1 # [ 138.167603] nix-daemon[855]: reaped child process 1471, status = succeeded lb # [ 138.148678] niks3-server[1213]: 2026/09/25 15:01:38 INFO Received push request method=POST path=/api/pushes worker1 # [ 138.180421] nix-grpc-daemon[1288]: time=2026-09-25T15:01:38.710Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 138.181968] nix-grpc-daemon[1288]: time=2026-09-25T15:01:38.710Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)" lb # [ 138.182466] niks3-server[1213]: 2026/09/25 15:01:38 INFO Received sign narinfos request method=POST path=/api/pushes/44/sign worker1 # [ 138.212113] nix-grpc-daemon[1288]: time=2026-09-25T15:01:38.743Z level=INFO msg="Uploading 1 narinfos" lb # [ 138.190505] niks3-server[1213]: 2026/09/25 15:01:38 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls lb # [ 138.197050] niks3-server[1213]: 2026/09/25 15:01:38 INFO Signed narinfos id=44 count=1 lb # [ 138.198715] niks3-server[1213]: 2026/09/25 15:01:38 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst lb # [ 138.209475] niks3-server[1213]: 2026/09/25 15:01:38 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker1 # [ 138.236403] nix-grpc-daemon[1288]: time=2026-09-25T15:01:38.768Z level=INFO msg="Upload complete. (219ms)" lb # [ 138.212435] niks3-server[1213]: 2026/09/25 15:01:38 INFO Received complete push request method=POST path=/api/pushes/44/complete lb # [ 138.216385] niks3-server[1213]: 2026/09/25 15:01:38 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo worker1 # [ 138.245021] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=1 worker1 # [ 138.258124] nix-daemon[855]: reaped child process 1450, status = succeeded worker1 # [ 138.260847] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:38Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=0 worker1 # [ 138.266072] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:38Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 138.270909] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 138.275143] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 138.281661] nix-grpc-daemon[1288]: time=2026-09-25T15:01:38.813Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'... worker1 # [ 138.288853] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 lb # [ 138.269897] envoy[897]: conn peer=[2001:db8:1::4]:52164 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=12628 tx=764 ms=13469 worker1 # [ 138.297620] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 138.418656] nix-daemon[855]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 138.440274] nix-daemon[855]: reaped child process 1480, status = succeeded lb # [ 138.420453] niks3-server[1213]: 2026/09/25 15:01:38 INFO Received push request method=POST path=/api/pushes worker1 # [ 138.450176] nix-grpc-daemon[1288]: time=2026-09-25T15:01:38.981Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 138.451759] nix-grpc-daemon[1288]: time=2026-09-25T15:01:38.981Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)" lb # [ 138.441444] niks3-server[1213]: 2026/09/25 15:01:38 INFO Received sign narinfos request method=POST path=/api/pushes/45/sign worker1 # [ 138.468293] nix-grpc-daemon[1288]: time=2026-09-25T15:01:39.000Z level=INFO msg="Uploading 1 narinfos" lb # [ 138.448122] niks3-server[1213]: 2026/09/25 15:01:38 INFO Signed narinfos id=45 count=1 lb # [ 138.450307] niks3-server[1213]: 2026/09/25 15:01:38 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst lb # [ 138.461109] niks3-server[1213]: 2026/09/25 15:01:38 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls lb # [ 138.463725] niks3-server[1213]: 2026/09/25 15:01:38 INFO Received complete push request method=POST path=/api/pushes/45/complete worker1 # [ 138.493668] nix-grpc-daemon[1288]: time=2026-09-25T15:01:39.025Z level=INFO msg="Upload complete. (185ms)" lb # [ 138.471366] niks3-server[1213]: 2026/09/25 15:01:38 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo worker1 # [ 138.500173] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=0 paths=1 nar_bytes_in=768 worker1 # [ 138.509560] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 138.514219] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 138.517891] nix-daemon[855]: reaped child process 1475, status = succeeded client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051... client # farm-top> node-a: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker1 # [ 138.528045] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 139.064108] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:39Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:38244 client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-on-standby worker1 # [ 139.351510] nix-daemon[855]: reaped child process 1484, status = succeeded worker1 # [ 139.438297] nix-daemon[855]: accepted connection from pid 1500, user nix-grpc-daemon (trusted) worker1 # [ 139.455888] nix-daemon[855]: reaped child process 1503, status = succeeded lb # [ 139.436759] niks3-server[1213]: 2026/09/25 15:01:39 INFO Received push request method=POST path=/api/pushes worker1 # [ 139.474890] nix-grpc-daemon[1288]: time=2026-09-25T15:01:40.001Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 139.478410] nix-grpc-daemon[1288]: time=2026-09-25T15:01:40.001Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)" lb # [ 139.473206] niks3-server[1213]: 2026/09/25 15:01:39 INFO Received sign narinfos request method=POST path=/api/pushes/46/sign worker1 # [ 139.505833] nix-grpc-daemon[1288]: time=2026-09-25T15:01:40.037Z level=INFO msg="Uploading 1 narinfos" lb # [ 139.482040] niks3-server[1213]: 2026/09/25 15:01:39 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv lb # [ 139.485004] niks3-server[1213]: 2026/09/25 15:01:40 INFO Signed narinfos id=46 count=1 lb # [ 139.494375] niks3-server[1213]: 2026/09/25 15:01:40 INFO Received complete push request method=POST path=/api/pushes/46/complete lb # [ 139.505147] niks3-server[1213]: 2026/09/25 15:01:40 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst lb # [ 139.512259] niks3-server[1213]: 2026/09/25 15:01:40 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls worker1 # [ 139.540384] nix-grpc-daemon[1288]: time=2026-09-25T15:01:40.072Z level=INFO msg="Upload complete. (204ms)" lb # [ 139.519118] niks3-server[1213]: 2026/09/25 15:01:40 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo worker1 # [ 139.548554] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=1 worker1 # [ 139.557679] nix-daemon[855]: reaped child process 1482, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 4.56 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.16 seconds) worker2 # [ 139.757427] systemd[1]: Starting scheduler-token.service... worker2 # [ 139.808092] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 139.812123] systemd[1]: Finished scheduler-token.service. worker2 # [ 139.813055] systemd[1]: scheduler-token.service: Consumed 25ms CPU time over 52ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 139.817804] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 139.941249] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:40Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 139.944845] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:40Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 139.949295] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:40Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 139.980576] nix-grpc-daemon[1372]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 139.982899] nix-grpc-daemon[1372]: E0000 00:00:1790348500.497272 1372 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 140.008385] nix-grpc-daemon[1372]: E0000 00:00:1790348500.499531 1372 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 140.012724] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:40Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 140.015496] nix-grpc-daemon[1372]: E0000 00:00:1790348500.507064 1412 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 140.019265] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 140.024420] nix-grpc-daemon[1372]: E0000 00:00:1790348500.533352 1412 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 140.028317] nix-grpc-daemon[1372]: E0000 00:00:1790348500.533521 1412 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 140.034512] nix-grpc-daemon[1372]: E0000 00:00:1790348500.549545 1412 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 140.038302] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:40Z level=info event=scheduler_connected addr=lb:50051 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1 # [ 140.088400] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:40Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 140.096371] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:40Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:38230 worker1 # [ 140.101506] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:40Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.18 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.42 seconds) subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs client # [ 140.417502] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs. client # Running as unit: rs.service; invocation ID: 62639a92d1a74bc8bd5610aff662ded7 client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.05 seconds) worker1 # [ 140.646464] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:33728 client # [ 140.630335] nix[1354]: don't know how to build these paths: client # [ 140.631285] nix[1354]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker1 # [ 140.662469] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:41Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=0 worker1 # [ 140.664824] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:41Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 140.677084] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 140.690658] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:33728 client # [ 140.676223] nix[1354]: copying 1 paths to 'lb:50051'... worker1 # [ 140.699550] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 client # [ 140.679127] nix[1354]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'... worker1 # [ 140.709584] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 140.834672] nix-daemon[855]: accepted connection from pid 1516, user nix-grpc-daemon (trusted) worker1 # [ 140.852475] nix-daemon[855]: reaped child process 1519, status = succeeded lb # [ 140.834723] niks3-server[1213]: 2026/09/25 15:01:41 INFO Received push request method=POST path=/api/pushes worker1 # [ 140.863194] nix-grpc-daemon[1288]: time=2026-09-25T15:01:41.394Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 140.865723] nix-grpc-daemon[1288]: time=2026-09-25T15:01:41.394Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)" lb # [ 140.854779] niks3-server[1213]: 2026/09/25 15:01:41 INFO Received sign narinfos request method=POST path=/api/pushes/47/sign worker1 # [ 140.885114] nix-grpc-daemon[1288]: time=2026-09-25T15:01:41.417Z level=INFO msg="Uploading 1 narinfos" lb # [ 140.863462] niks3-server[1213]: 2026/09/25 15:01:41 INFO Signed narinfos id=47 count=1 lb # [ 140.867917] niks3-server[1213]: 2026/09/25 15:01:41 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls lb # [ 140.876651] niks3-server[1213]: 2026/09/25 15:01:41 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst lb # [ 140.881949] niks3-server[1213]: 2026/09/25 15:01:41 INFO Received complete push request method=POST path=/api/pushes/47/complete worker1 # [ 140.910652] nix-grpc-daemon[1288]: time=2026-09-25T15:01:41.442Z level=INFO msg="Upload complete. (191ms)" lb # [ 140.887583] niks3-server[1213]: 2026/09/25 15:01:41 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo worker1 # [ 140.916191] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 140.923292] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 worker1 # [ 140.932152] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) worker1 # [ 140.936072] nix-daemon[855]: reaped child process 1514, status = succeeded client # [ 140.922020] nix[1354]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051... worker1 # [ 140.947511] nix-daemon[855]: accepted connection from pid 1226, user nix-grpc-daemon (trusted) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true worker2 # [ 142.974930] systemd-journald[272]: Received client request to rotate journal, rotating. worker2 # [ 142.986270] systemd-journald[272]: Vacuuming done, freed 0B of archived journals from /var/log/journal/0530605f59a340c7bb6a90097d89d276. worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.04 seconds) worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service worker1 # [ 143.035702] systemd[1]: Stopping Nix worker-protocol over gRPC... worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds) client: must succeed: sleep 5.0 worker1 # [ 143.313890] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:43Z level=info event=draining client: (finished: must succeed: sleep 5.0, in 5.03 seconds) worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=' lb # [ 148.268134] envoy[897]: conn peer=[2001:db8:1::1]:55064 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9602 tx=4346 ms=4379 worker1 # [ 148.431423] nix-daemon[855]: reaped child process 1523, status = succeeded worker1 # [ 148.494379] nix-daemon[855]: accepted connection from pid 1564, user nix-grpc-daemon (trusted) worker1 # [ 148.510162] nix-daemon[855]: reaped child process 1567, status = succeeded lb # [ 148.492217] niks3-server[1213]: 2026/09/25 15:01:49 INFO Received push request method=POST path=/api/pushes worker1 # [ 148.524260] nix-grpc-daemon[1288]: time=2026-09-25T15:01:49.053Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 148.525754] nix-grpc-daemon[1288]: time=2026-09-25T15:01:49.053Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)" worker1 # [ 148.545580] nix-grpc-daemon[1288]: time=2026-09-25T15:01:49.077Z level=INFO msg="Uploading 1 narinfos" lb # [ 148.522805] niks3-server[1213]: 2026/09/25 15:01:49 INFO Received sign narinfos request method=POST path=/api/pushes/48/sign lb # [ 148.528932] niks3-server[1213]: 2026/09/25 15:01:49 INFO Signed narinfos id=48 count=1 lb # [ 148.530692] niks3-server[1213]: 2026/09/25 15:01:49 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv lb # [ 148.540691] niks3-server[1213]: 2026/09/25 15:01:49 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst worker1 # [ 148.574676] nix-grpc-daemon[1288]: time=2026-09-25T15:01:49.106Z level=INFO msg="Upload complete. (214ms)" lb # [ 148.551134] niks3-server[1213]: 2026/09/25 15:01:49 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls lb # [ 148.557423] niks3-server[1213]: 2026/09/25 15:01:49 INFO Received complete push request method=POST path=/api/pushes/48/complete lb # [ 148.560423] niks3-server[1213]: 2026/09/25 15:01:49 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo worker1 # [ 148.584687] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33728 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=1 worker1 # [ 148.594360] nix-daemon[855]: reaped child process 1521, status = succeeded client # [ 148.585718] systemd[1]: rs.service: Deactivated successfully. client # [ 148.588782] systemd[1]: rs.service: Consumed 173ms CPU time over 8.170s wall clock time, 41.4M memory peak, 6.9K incoming IP traffic, 8.3K outgoing IP traffic. worker1 # [ 149.315579] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:49Z level=info event=signal_exit lb # [ 149.513283] niks3-server[1213]: 2026/09/25 15:01:50 INFO lead: released remote=192.168.1.3:37636 lb # [ 149.976992] niks3-server[1213]: 2026/09/25 15:01:50 INFO lead: acquired remote=192.168.1.4:50188 worker2 # [ 149.976218] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:50Z level=info event=scheduler_take_over worker1 # [ 150.533894] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:51Z level=info event=scheduler_restarting lb # [ 150.509679] envoy[897]: [2026-09-25 15:01:51.031][897][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream} worker2 # [ 150.511529] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:51Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 150.539743] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:51Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:38230 worker1 # [ 150.545957] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:51Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 150.548940] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:51Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 150.551226] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:51Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:38230 worker1 # [ 150.554732] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:51Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 150.569397] nix-daemon[855]: reaped child process 1278, status = succeeded worker2 # [ 150.712496] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:51Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 150.721637] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:51Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 150.723845] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:51Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:33270 worker2 # [ 150.727067] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:51Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 150.758433] nix-grpc-daemon[1226]: ts=2026-09-25T15:01:51Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 151.005080] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:51Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:58954 worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.14 seconds) client: waiting for success: ! systemctl is-active rs client: (finished: waiting for success: ! systemctl is-active rs, in 0.03 seconds) client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; } client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.04 seconds) client: must fail: journalctl -u rs | grep -q 'reconnecting' client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker1 # [ 152.170790] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 152.174720] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 152.177849] systemd[1]: nix-grpc-daemon.service: Consumed 2.000s CPU time over 31.007s wall clock time, 46.8M memory peak, 365.4K incoming IP traffic, 244.7K outgoing IP traffic. worker1 # [ 152.188078] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 152.299263] nix-grpc-daemon[1575]: ts=2026-09-25T15:01:52Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 152.301589] nix-grpc-daemon[1575]: ts=2026-09-25T15:01:52Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 152.305414] nix-grpc-daemon[1575]: ts=2026-09-25T15:01:52Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 152.311820] nix-grpc-daemon[1575]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 152.328105] nix-grpc-daemon[1575]: E0000 00:00:1790348512.841734 1575 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 152.353001] nix-grpc-daemon[1575]: E0000 00:00:1790348512.843732 1575 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 152.355821] nix-grpc-daemon[1575]: ts=2026-09-25T15:01:52Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 152.357756] nix-grpc-daemon[1575]: E0000 00:00:1790348512.851620 1616 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 152.363166] nix-grpc-daemon[1575]: E0000 00:00:1790348512.869567 1616 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 152.367570] nix-grpc-daemon[1575]: E0000 00:00:1790348512.869709 1616 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 152.341594] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:52Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:33270 worker1 # [ 152.370713] nix-grpc-daemon[1575]: E0000 00:00:1790348512.884115 1616 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 152.341799] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:52Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 152.375615] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 152.377582] nix-grpc-daemon[1575]: ts=2026-09-25T15:01:52Z level=info event=scheduler_connected addr=lb:50051 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.24 seconds) subtest: scheduler restart mid-build: build finishes, no double build client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid client # [ 152.664876] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid. client # Running as unit: mid.service; invocation ID: df07780cf0d740db81f7751a50c93175 client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.05 seconds) worker1 # [ 152.859663] nix-grpc-daemon[1575]: ts=2026-09-25T15:01:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 152.868295] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) client # [ 152.857355] nix[1390]: don't know how to build these paths: client # [ 152.858449] nix[1390]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker2 # [ 152.857501] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:53Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker2 # [ 152.864357] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:53Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 152.872628] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58958 worker2 # [ 152.879385] nix-daemon[845]: accepted connection from pid 1372, user nix-grpc-daemon (trusted) worker2 # [ 152.892300] nix-grpc-daemon[1372]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 70 ms (attempt 1/5) worker2 # [ 152.967525] nix-grpc-daemon[1372]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 114 ms (attempt 2/5) worker2 # [ 153.082402] nix-grpc-daemon[1372]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 60 ms (attempt 3/5) worker2 # [ 153.143323] nix-grpc-daemon[1372]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 662 ms (attempt 4/5) worker2 # [ 153.808816] nix-grpc-daemon[1372]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 153.811104] nix-grpc-daemon[1372]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 153.821975] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58958 client # [ 153.835999] nix[1390]: copying 1 paths to 'lb:50051'... worker2 # [ 153.830660] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58958 client # [ 153.836285] nix[1390]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'... worker2 # [ 153.840865] nix-daemon[845]: accepted connection from pid 1372, user nix-grpc-daemon (trusted) worker2 # [ 153.875223] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:54Z level=info event=niks3_push_started pid=1504 worker2 # [ 153.995057] nix-grpc-daemon[1504]: time=2026-09-25T15:01:54.537Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 154.100061] nix-daemon[845]: accepted connection from pid 1516, user nix-grpc-daemon (trusted) worker2 # [ 154.118483] nix-daemon[845]: reaped child process 1519, status = succeeded lb # [ 154.128672] niks3-server[1213]: 2026/09/25 15:01:54 INFO Received push request method=POST path=/api/pushes worker2 # [ 154.131005] nix-grpc-daemon[1504]: time=2026-09-25T15:01:54.673Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 154.136656] nix-grpc-daemon[1504]: time=2026-09-25T15:01:54.673Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)" lb # [ 154.155315] niks3-server[1213]: 2026/09/25 15:01:54 INFO Received sign narinfos request method=POST path=/api/pushes/49/sign worker2 # [ 154.159016] nix-grpc-daemon[1504]: time=2026-09-25T15:01:54.700Z level=INFO msg="Uploading 1 narinfos" lb # [ 154.164807] niks3-server[1213]: 2026/09/25 15:01:54 INFO Signed narinfos id=49 count=1 lb # [ 154.170650] niks3-server[1213]: 2026/09/25 15:01:54 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst lb # [ 154.180407] niks3-server[1213]: 2026/09/25 15:01:54 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls lb # [ 154.182860] niks3-server[1213]: 2026/09/25 15:01:54 INFO Received complete push request method=POST path=/api/pushes/49/complete worker2 # [ 154.184642] nix-grpc-daemon[1504]: time=2026-09-25T15:01:54.727Z level=INFO msg="Upload complete. (185ms)" lb # [ 154.189572] niks3-server[1213]: 2026/09/25 15:01:54 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo worker2 # [ 154.191314] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58958 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 154.196645] nix-grpc-daemon[1372]: ts=2026-09-25T15:01:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58958 worker2 # [ 154.206235] nix-daemon[845]: accepted connection from pid 1372, user nix-grpc-daemon (trusted) worker2 # [ 154.208274] nix-daemon[845]: reaped child process 1502, status = succeeded client # [ 154.221115] nix[1390]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... worker2 # [ 154.218031] nix-daemon[845]: accepted connection from pid 1372, user nix-grpc-daemon (trusted) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service worker2 # [ 156.306387] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1372 (nix-grpc-daemon) on client request. worker2 # [ 156.308463] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request. lb # [ 156.326593] niks3-server[1213]: 2026/09/25 15:01:56 INFO lead: released remote=192.168.1.4:50188 worker2 # [ 156.328876] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL client # [ 156.332910] nix[1390]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again lb # [ 156.331045] envoy[897]: [2026-09-25 15:01:56.851][897][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 13, upstream_reset_after_response_started{connection_termination} worker2 # [ 156.330533] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'. worker1 # [ 156.355150] nix-grpc-daemon[1575]: ts=2026-09-25T15:01:56Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination" client # [ 156.336304] nix[1390]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting worker2 # [ 156.331917] systemd[1]: nix-grpc-daemon.service: Consumed 428ms CPU time over 16.508s wall clock time, 34.8M memory peak, 216.7K incoming IP traffic, 52.7K outgoing IP traffic. worker2 # [ 156.355864] systemd[1]: Starting scheduler-token.service... worker2 # [ 156.390088] nix-daemon[845]: reaped child process 1522, status = succeeded worker2 # [ 156.391752] nix-daemon[845]: reaped child process 1492, status = succeeded worker2 # [ 156.404336] nix-daemon[1524]: unexpected Nix daemon error: error: interrupted by the user worker2 # [ 156.413008] nix-daemon[845]: reaped child process 1524, status = failed with exit code 1 worker2 # [ 156.422303] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 156.424881] systemd[1]: Finished scheduler-token.service. worker2 # [ 156.425728] systemd[1]: scheduler-token.service: Consumed 24ms CPU time over 76ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 156.431894] systemd[1]: nix-grpc-daemon.service: Scheduled restart job, restart counter is at 1. worker2 # [ 156.437269] systemd[1]: Starting scheduler-token.service... worker2 # [ 156.472903] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 156.474689] systemd[1]: Finished scheduler-token.service. worker2 # [ 156.478171] systemd[1]: scheduler-token.service: Consumed 22ms CPU time over 43ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 156.486101] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 156.555243] nix-grpc-daemon[1575]: ts=2026-09-25T15:01:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 156.589359] nix-grpc-daemon[1575]: ts=2026-09-25T15:01:57Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure" worker2 # [ 156.581457] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 156.584158] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 156.588079] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 156.594877] nix-grpc-daemon[1567]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR lb # [ 156.609063] niks3-server[1213]: 2026/09/25 15:01:57 INFO lead: acquired remote=192.168.1.4:52130 worker2 # [ 156.623441] nix-grpc-daemon[1567]: E0000 00:00:1790348517.135189 1567 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.37 seconds) client: must succeed: sleep 5.0 worker2 # [ 156.644959] nix-grpc-daemon[1567]: E0000 00:00:1790348517.137283 1567 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 156.648696] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 156.650465] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=info event=scheduler_take_over worker2 # [ 156.654989] nix-grpc-daemon[1567]: E0000 00:00:1790348517.148443 1607 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 156.662559] nix-grpc-daemon[1567]: E0000 00:00:1790348517.169268 1607 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 156.666203] nix-grpc-daemon[1567]: E0000 00:00:1790348517.169845 1607 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 156.670450] nix-grpc-daemon[1567]: E0000 00:00:1790348517.186342 1607 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 156.673724] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 156.676638] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 156.678024] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 156.679443] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:59030 worker2 # [ 156.681310] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 156.790437] nix-grpc-daemon[1575]: ts=2026-09-25T15:01:57Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 156.765022] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:59030 worker2 # [ 156.767876] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker2 # [ 156.836172] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker2 # [ 156.843555] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 156.850079] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59038 worker2 # [ 156.858115] nix-daemon[845]: accepted connection from pid 1567, user nix-grpc-daemon (trusted) worker2 # [ 156.871673] nix-daemon[845]: accepted connection from pid 1567, user nix-grpc-daemon (trusted) client # [ 156.885715] nix[1390]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... worker2 # [ 156.881920] nix-daemon[845]: accepted connection from pid 1567, user nix-grpc-daemon (trusted) worker2 # [ 157.963516] nix-grpc-daemon[1567]: ts=2026-09-25T15:01:58Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:59074 lb # [ 158.272872] envoy[897]: conn peer=[2001:db8:1::1]:55076 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2962 tx=1497 ms=7954 lb # [ 158.275548] envoy[897]: conn peer=[2001:db8:1::3]:39150 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=5454 tx=1760 ms=29394 lb # [ 158.278276] envoy[897]: conn peer=[2001:db8:1::4]:60622 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=1961 tx=328 ms=16310 client: (finished: must succeed: sleep 5.0, in 5.03 seconds) client: waiting for success: ! systemctl is-active mid worker2 # [ 161.902419] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:02Z level=info event=niks3_push_started pid=1650 worker2 # [ 161.922887] nix-daemon[845]: reaped child process 1625, status = succeeded worker2 # [ 162.021672] nix-grpc-daemon[1650]: time=2026-09-25T15:02:02.563Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 162.123414] nix-daemon[845]: accepted connection from pid 1662, user nix-grpc-daemon (trusted) worker2 # [ 162.144364] nix-daemon[845]: reaped child process 1665, status = succeeded lb # [ 162.146904] niks3-server[1213]: 2026/09/25 15:02:02 INFO Received push request method=POST path=/api/pushes worker2 # [ 162.151948] nix-grpc-daemon[1650]: time=2026-09-25T15:02:02.692Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 162.155756] nix-grpc-daemon[1650]: time=2026-09-25T15:02:02.692Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)" lb # [ 162.183104] niks3-server[1213]: 2026/09/25 15:02:02 INFO Received sign narinfos request method=POST path=/api/pushes/50/sign lb # [ 162.194873] niks3-server[1213]: 2026/09/25 15:02:02 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst lb # [ 162.206841] niks3-server[1213]: 2026/09/25 15:02:02 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker2 # [ 162.212117] nix-grpc-daemon[1650]: time=2026-09-25T15:02:02.753Z level=INFO msg="Uploading 1 narinfos" lb # [ 162.217892] niks3-server[1213]: 2026/09/25 15:02:02 INFO Signed narinfos id=50 count=1 lb # [ 162.227821] niks3-server[1213]: 2026/09/25 15:02:02 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls lb # [ 162.236734] niks3-server[1213]: 2026/09/25 15:02:02 INFO Received complete push request method=POST path=/api/pushes/50/complete lb # [ 162.248479] niks3-server[1213]: 2026/09/25 15:02:02 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo worker2 # [ 162.248772] nix-grpc-daemon[1650]: time=2026-09-25T15:02:02.791Z level=INFO msg="Upload complete. (223ms)" worker2 # [ 162.260123] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59038 duration_s=5 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=1 worker2 # [ 162.269277] nix-daemon[845]: reaped child process 1623, status = succeeded client # [ 162.284626] systemd[1]: mid.service: Deactivated successfully. client # [ 162.287382] systemd[1]: mid.service: Consumed 173ms CPU time over 9.621s wall clock time, 41.2M memory peak, 8.6K incoming IP traffic, 10.4K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active mid, in 1.07 seconds) client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; } client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.04 seconds) (finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.34 seconds) subtest: interrupting the client stops the build on the worker client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr client # [ 163.001823] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr. client # Running as unit: intr.service; invocation ID: 8d9c47fa70ca44148e392e3055ed805f client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.04 seconds) worker1 # [ 163.179888] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:52466 client # [ 163.162298] nix[1426]: don't know how to build these paths: client # [ 163.163268] nix[1426]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker2 # [ 163.166041] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:03Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=0 worker2 # [ 163.169493] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:03Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 163.199021] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 163.207569] nix-grpc-daemon[1575]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 66 ms (attempt 1/5) worker1 # [ 163.278583] nix-grpc-daemon[1575]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 82 ms (attempt 2/5) worker1 # [ 163.361622] nix-grpc-daemon[1575]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 302 ms (attempt 3/5) worker1 # [ 163.664705] nix-grpc-daemon[1575]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 444 ms (attempt 4/5) worker1 # [ 164.110482] nix-grpc-daemon[1575]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 164.113774] nix-grpc-daemon[1575]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 164.126476] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 client # [ 164.112135] nix[1426]: copying 1 paths to 'lb:50051'... worker1 # [ 164.135619] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 client # [ 164.115128] nix[1426]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'... worker1 # [ 164.147590] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 164.176462] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:04Z level=info event=niks3_push_started pid=1671 worker1 # [ 164.305491] nix-grpc-daemon[1671]: time=2026-09-25T15:02:04.836Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 164.403545] nix-daemon[855]: accepted connection from pid 1683, user nix-grpc-daemon (trusted) worker1 # [ 164.421339] nix-daemon[855]: reaped child process 1686, status = succeeded lb # [ 164.401506] niks3-server[1213]: 2026/09/25 15:02:04 INFO Received push request method=POST path=/api/pushes worker1 # [ 164.433715] nix-grpc-daemon[1671]: time=2026-09-25T15:02:04.964Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 164.435270] nix-grpc-daemon[1671]: time=2026-09-25T15:02:04.964Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)" lb # [ 164.428886] niks3-server[1213]: 2026/09/25 15:02:04 INFO Received sign narinfos request method=POST path=/api/pushes/51/sign worker1 # [ 164.456878] nix-grpc-daemon[1671]: time=2026-09-25T15:02:04.988Z level=INFO msg="Uploading 1 narinfos" lb # [ 164.437046] niks3-server[1213]: 2026/09/25 15:02:04 INFO Signed narinfos id=51 count=1 lb # [ 164.441462] niks3-server[1213]: 2026/09/25 15:02:04 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst lb # [ 164.448969] niks3-server[1213]: 2026/09/25 15:02:04 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls lb # [ 164.460131] niks3-server[1213]: 2026/09/25 15:02:04 INFO Received complete push request method=POST path=/api/pushes/51/complete lb # [ 164.467783] niks3-server[1213]: 2026/09/25 15:02:04 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo worker1 # [ 164.492669] nix-grpc-daemon[1671]: time=2026-09-25T15:02:05.023Z level=INFO msg="Upload complete. (182ms)" worker1 # [ 164.495247] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=424 worker1 # [ 164.509798] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 164.514503] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 164.518600] nix-daemon[855]: reaped child process 1668, status = succeeded worker1 # [ 164.527519] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) client # [ 164.504963] nix[1426]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051... client: must succeed: systemctl kill -s INT intr client # [ 166.505015] systemd[1]: intr.service: Sent signal SIGINT to main process 1426 (nix) on client request. worker1 # [ 166.531287] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:07Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv client # [ 166.509731] nix[1426]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client # [ 166.517767] nix[1426]: error: interrupted by the user client: (finished: must succeed: systemctl kill -s INT intr, in 0.04 seconds) client # [ 166.536109] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE client # [ 166.538209] systemd[1]: intr.service: Failed with result 'exit-code'. worker1 # [ 166.565937] nix-daemon[1690]: unexpected Nix daemon error: error: interrupted by the user client # [ 166.541905] systemd[1]: intr.service: Consumed 150ms CPU time over 3.538s wall clock time, 41.4M memory peak, 6.4K incoming IP traffic, 8.2K outgoing IP traffic. worker1 # [ 166.576325] nix-daemon[855]: reaped child process 1688, status = succeeded worker1 # [ 166.585311] nix-daemon[855]: reaped child process 1690, status = failed with exit code 1 (finished: subtest: interrupting the client stops the build on the worker, in 3.64 seconds) subtest: a deploy mid-build drains: the build finishes, the new generation takes over client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw client # [ 166.648109] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw. client # Running as unit: sw.service; invocation ID: ea7ddf7a3c9d453aa87ee4897d842918 client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.05 seconds) worker2 # [ 166.817687] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59038 client # [ 166.827926] nix[1447]: don't know how to build these paths: client # [ 166.829017] nix[1447]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker2 # [ 166.830511] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:07Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=0 worker2 # [ 166.834458] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:07Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 166.841798] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59038 worker2 # [ 166.846711] nix-grpc-daemon[1567]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 34 ms (attempt 1/5) worker2 # [ 166.881642] nix-grpc-daemon[1567]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 169 ms (attempt 2/5) worker2 # [ 167.051748] nix-grpc-daemon[1567]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 52 ms (attempt 3/5) worker2 # [ 167.104712] nix-grpc-daemon[1567]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 563 ms (attempt 4/5) worker2 # [ 167.669076] nix-grpc-daemon[1567]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 167.671093] nix-grpc-daemon[1567]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 167.683069] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59038 client # [ 167.696093] nix[1447]: copying 1 paths to 'lb:50051'... worker2 # [ 167.691844] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59038 client # [ 167.699059] nix[1447]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'... worker2 # [ 167.700609] nix-daemon[845]: accepted connection from pid 1567, user nix-grpc-daemon (trusted) worker2 # [ 167.838273] nix-daemon[845]: accepted connection from pid 1701, user nix-grpc-daemon (trusted) lb # [ 167.865880] niks3-server[1213]: 2026/09/25 15:02:08 INFO Received push request method=POST path=/api/pushes worker2 # [ 167.872419] nix-grpc-daemon[1650]: time=2026-09-25T15:02:08.411Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 167.873936] nix-grpc-daemon[1650]: time=2026-09-25T15:02:08.411Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)" worker2 # [ 167.880117] nix-daemon[845]: reaped child process 1707, status = succeeded lb # [ 167.886616] niks3-server[1213]: 2026/09/25 15:02:08 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst worker2 # [ 167.888158] nix-grpc-daemon[1650]: time=2026-09-25T15:02:08.430Z level=INFO msg="Uploading 1 narinfos" lb # [ 167.894879] niks3-server[1213]: 2026/09/25 15:02:08 INFO Received sign narinfos request method=POST path=/api/pushes/52/sign lb # [ 167.900896] niks3-server[1213]: 2026/09/25 15:02:08 INFO Signed narinfos id=52 count=1 lb # [ 167.902640] niks3-server[1213]: 2026/09/25 15:02:08 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls lb # [ 167.913069] niks3-server[1213]: 2026/09/25 15:02:08 INFO Received complete push request method=POST path=/api/pushes/52/complete worker2 # [ 167.914168] nix-grpc-daemon[1650]: time=2026-09-25T15:02:08.456Z level=INFO msg="Upload complete. (204ms)" lb # [ 167.918649] niks3-server[1213]: 2026/09/25 15:02:08 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo worker2 # [ 167.924915] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59038 duration_s=0 paths=1 nar_bytes_in=416 worker2 # [ 167.927499] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59038 worker2 # [ 167.931003] nix-daemon[845]: reaped child process 1699, status = succeeded worker2 # [ 167.934839] nix-daemon[845]: accepted connection from pid 1567, user nix-grpc-daemon (trusted) client # [ 167.947704] nix[1447]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051... worker2 # [ 167.944231] nix-daemon[845]: accepted connection from pid 1567, user nix-grpc-daemon (trusted) lb # [ 168.272858] envoy[897]: conn peer=[2001:db8:1::1]:50044 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3927 tx=1941 ms=9437 lb # [ 168.275534] envoy[897]: conn peer=[2001:db8:1::1]:37262 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2974 tx=825 ms=3366 worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2 worker2 # Checking switch inhibitors... done worker2 # [ 169.455154] nixos[1740]: switching to system configuration /nix/store/psv6z7q5q9yl9akmyzsis8bxmmshihgf-nixos-system-worker2-test worker2 # activating the configuration... worker2 # [ 169.462427] systemd[1]: Stopped target Local File Systems. worker2 # [ 169.463285] systemd[1]: Stopped target Remote File Systems. worker2 # setting up /etc... worker2 # [ 169.976696] systemd[1]: Reload requested from client PID 1740 ('.switch-to-conf') (unit backdoor.service)... worker2 # [ 169.978234] systemd[1]: Reloading... worker2 # [ 174.636623] systemd[1]: Reloading finished in 4646 ms. worker2 # restarting sysinit-reactivation.target worker2 # [ 174.763083] systemd[1]: Starting Re-setup tmpfiles on a system that is already running.... worker2 # [ 174.886745] systemd[1]: Finished Re-setup tmpfiles on a system that is already running.. worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service[ 174.890020] systemd[1]: Reached target Reactivate sysinit units. worker2 # worker2 # [ 174.892670] systemd[1]: Reloading D-Bus System Message Bus... worker2 # [ 174.893837] dbus-broker-launch[492]: Caught SIGHUP, trigger reload. worker2 # [ 174.905759] systemd[1]: Reloading Nix worker-protocol over gRPC... worker2 # [ 174.918106] dbus-broker-launch[492]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 174.928701] dbus-broker-launch[492]: NSS returned no entry for 'systemd-timesync' worker2 # [ 174.929744] dbus-broker-launch[492]: Invalid user-name in /nix/store/ng2ydfkv03gvmdws4f4mjf9wj6qj3kq6-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker2 # [ 174.942590] systemd[1]: Reloaded Nix worker-protocol over gRPC. worker2 # [ 174.954811] systemd[1]: Reloaded D-Bus System Message Bus. worker2 # [ 174.961627] systemd[1]: Reached target Local File Systems. worker2 # [ 174.964758] systemd[1]: Reached target Remote File Systems. worker2 # [ 174.974042] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker2 # [ 174.975565] systemd[1]: Starting Create SUID/SGID Wrappers... worker2 # [ 174.985166] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker2 # [ 174.986477] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker2 # [ 174.996105] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 174.997714] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 174.998983] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 175.003423] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 175.005647] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 175.300544] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 175.301661] systemd[1]: Finished Create SUID/SGID Wrappers. worker2 # [ 175.308468] systemd[1]: Starting linger-users.service... worker2 # [ 175.350364] systemd[1]: linger-users.service: Deactivated successfully. worker2 # [ 175.351381] systemd[1]: Finished linger-users.service. worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service worker2 # [ 175.604266] nixos[1740]: finished switching to system configuration /nix/store/psv6z7q5q9yl9akmyzsis8bxmmshihgf-nixos-system-worker2-test worker2 # [ 175.607457] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:16Z level=info event=draining worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 6.55 seconds) worker2: must succeed: pgrep -f 'read -t [9]0 x' worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2 worker1 # [ 175.878127] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:16Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:52466 client # don't know how to build these paths: client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker1 # [ 175.885451] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker2 # [ 175.873122] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:16Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=0 worker2 # [ 175.875263] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:16Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 175.904709] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 175.912924] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 client # copying 1 paths to 'lb:50051'... worker1 # [ 175.921492] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'... worker1 # [ 175.932586] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 176.062389] nix-daemon[855]: accepted connection from pid 1737, user nix-grpc-daemon (trusted) worker1 # [ 176.078028] nix-daemon[855]: reaped child process 1740, status = succeeded lb # [ 176.061050] niks3-server[1213]: 2026/09/25 15:02:16 INFO Received push request method=POST path=/api/pushes worker1 # [ 176.091853] nix-grpc-daemon[1671]: time=2026-09-25T15:02:16.623Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 176.093537] nix-grpc-daemon[1671]: time=2026-09-25T15:02:16.623Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)" lb # [ 176.078573] niks3-server[1213]: 2026/09/25 15:02:16 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst worker1 # [ 176.109767] nix-grpc-daemon[1671]: time=2026-09-25T15:02:16.641Z level=INFO msg="Uploading 1 narinfos" lb # [ 176.086687] niks3-server[1213]: 2026/09/25 15:02:16 INFO Received sign narinfos request method=POST path=/api/pushes/53/sign lb # [ 176.093026] niks3-server[1213]: 2026/09/25 15:02:16 INFO Signed narinfos id=53 count=1 lb # [ 176.097316] niks3-server[1213]: 2026/09/25 15:02:16 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls lb # [ 176.106921] niks3-server[1213]: 2026/09/25 15:02:16 INFO Received complete push request method=POST path=/api/pushes/53/complete worker1 # [ 176.135319] nix-grpc-daemon[1671]: time=2026-09-25T15:02:16.667Z level=INFO msg="Upload complete. (191ms)" lb # [ 176.112740] niks3-server[1213]: 2026/09/25 15:02:16 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo worker1 # [ 176.141522] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=528 worker1 # [ 176.151511] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 176.156494] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 176.160649] nix-daemon[855]: reaped child process 1735, status = succeeded client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051... client # farm-a> node-a: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker1 # [ 176.170879] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-during-drain worker1 # [ 177.431736] nix-daemon[855]: reaped child process 1744, status = succeeded worker1 # [ 177.489210] nix-daemon[855]: accepted connection from pid 1760, user nix-grpc-daemon (trusted) worker1 # [ 177.510256] nix-daemon[855]: reaped child process 1763, status = succeeded lb # [ 177.484947] niks3-server[1213]: 2026/09/25 15:02:18 INFO Received push request method=POST path=/api/pushes worker1 # [ 177.516378] nix-grpc-daemon[1671]: time=2026-09-25T15:02:18.047Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 177.519670] nix-grpc-daemon[1671]: time=2026-09-25T15:02:18.047Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)" lb # [ 177.507706] niks3-server[1213]: 2026/09/25 15:02:18 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst lb # [ 177.517584] niks3-server[1213]: 2026/09/25 15:02:18 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv lb # [ 177.524576] niks3-server[1213]: 2026/09/25 15:02:18 INFO Received sign narinfos request method=POST path=/api/pushes/54/sign worker1 # [ 177.552293] nix-grpc-daemon[1671]: time=2026-09-25T15:02:18.084Z level=INFO msg="Uploading 1 narinfos" lb # [ 177.530977] niks3-server[1213]: 2026/09/25 15:02:18 INFO Signed narinfos id=54 count=1 lb # [ 177.534589] niks3-server[1213]: 2026/09/25 15:02:18 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls lb # [ 177.543317] niks3-server[1213]: 2026/09/25 15:02:18 INFO Received complete push request method=POST path=/api/pushes/54/complete worker1 # [ 177.571910] nix-grpc-daemon[1671]: time=2026-09-25T15:02:18.103Z level=INFO msg="Upload complete. (193ms)" lb # [ 177.549221] niks3-server[1213]: 2026/09/25 15:02:18 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo worker2 # [ 177.553944] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:18Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=0 worker1 # [ 177.580616] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=1 worker2 # [ 177.556038] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:18Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 177.590854] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 177.594805] nix-daemon[855]: reaped child process 1742, status = succeeded worker1 # [ 177.599927] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 177.607982] nix-grpc-daemon[1671]: time=2026-09-25T15:02:18.140Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker1 # [ 177.614117] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'... worker1 # [ 177.627529] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 177.745934] nix-daemon[855]: accepted connection from pid 1767, user nix-grpc-daemon (trusted) worker1 # [ 177.764448] nix-daemon[855]: reaped child process 1770, status = succeeded lb # [ 177.742322] niks3-server[1213]: 2026/09/25 15:02:18 INFO Received push request method=POST path=/api/pushes worker1 # [ 177.771016] nix-grpc-daemon[1671]: time=2026-09-25T15:02:18.302Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 177.772717] nix-grpc-daemon[1671]: time=2026-09-25T15:02:18.302Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)" lb # [ 177.758772] niks3-server[1213]: 2026/09/25 15:02:18 INFO Received sign narinfos request method=POST path=/api/pushes/55/sign worker1 # [ 177.785455] nix-grpc-daemon[1671]: time=2026-09-25T15:02:18.317Z level=INFO msg="Uploading 1 narinfos" lb # [ 177.765589] niks3-server[1213]: 2026/09/25 15:02:18 INFO Signed narinfos id=55 count=1 lb # [ 177.770386] niks3-server[1213]: 2026/09/25 15:02:18 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls lb # [ 177.779657] niks3-server[1213]: 2026/09/25 15:02:18 INFO Received complete push request method=POST path=/api/pushes/55/complete worker1 # [ 177.808425] nix-grpc-daemon[1671]: time=2026-09-25T15:02:18.340Z level=INFO msg="Upload complete. (171ms)" lb # [ 177.786492] niks3-server[1213]: 2026/09/25 15:02:18 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst worker1 # [ 177.815067] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=640 lb # [ 177.791831] niks3-server[1213]: 2026/09/25 15:02:18 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo worker1 # [ 177.823697] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 177.830736] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 177.832481] nix-daemon[855]: reaped child process 1765, status = succeeded client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051... client # farm-b> node-a: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker1 # [ 177.842189] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-during-drain worker1 # [ 179.080448] nix-daemon[855]: reaped child process 1774, status = succeeded worker1 # [ 179.170781] nix-daemon[855]: accepted connection from pid 1790, user nix-grpc-daemon (trusted) worker1 # [ 179.188497] nix-daemon[855]: reaped child process 1793, status = succeeded lb # [ 179.169715] niks3-server[1213]: 2026/09/25 15:02:19 INFO Received push request method=POST path=/api/pushes worker1 # [ 179.199337] nix-grpc-daemon[1671]: time=2026-09-25T15:02:19.730Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 179.200976] nix-grpc-daemon[1671]: time=2026-09-25T15:02:19.730Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)" lb # [ 179.192727] niks3-server[1213]: 2026/09/25 15:02:19 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker1 # [ 179.224802] nix-grpc-daemon[1671]: time=2026-09-25T15:02:19.756Z level=INFO msg="Uploading 1 narinfos" lb # [ 179.202595] niks3-server[1213]: 2026/09/25 15:02:19 INFO Received sign narinfos request method=POST path=/api/pushes/56/sign lb # [ 179.208295] niks3-server[1213]: 2026/09/25 15:02:19 INFO Signed narinfos id=56 count=1 lb # [ 179.209958] niks3-server[1213]: 2026/09/25 15:02:19 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls lb # [ 179.220514] niks3-server[1213]: 2026/09/25 15:02:19 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst worker1 # [ 179.247752] nix-grpc-daemon[1671]: time=2026-09-25T15:02:19.779Z level=INFO msg="Upload complete. (198ms)" lb # [ 179.223713] niks3-server[1213]: 2026/09/25 15:02:19 INFO Received complete push request method=POST path=/api/pushes/56/complete lb # [ 179.228674] niks3-server[1213]: 2026/09/25 15:02:19 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo worker1 # [ 179.256610] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=1 worker2 # [ 179.232288] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:19Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=0 worker2 # [ 179.234343] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:19Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 179.267286] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 179.271393] nix-daemon[855]: reaped child process 1772, status = succeeded worker1 # [ 179.276698] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 179.284882] nix-grpc-daemon[1671]: time=2026-09-25T15:02:19.816Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'... worker1 # [ 179.291377] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 179.300956] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 179.416140] nix-daemon[855]: accepted connection from pid 1797, user nix-grpc-daemon (trusted) worker1 # [ 179.432534] nix-daemon[855]: reaped child process 1800, status = succeeded lb # [ 179.413346] niks3-server[1213]: 2026/09/25 15:02:19 INFO Received push request method=POST path=/api/pushes worker1 # [ 179.441623] nix-grpc-daemon[1671]: time=2026-09-25T15:02:19.973Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 179.443183] nix-grpc-daemon[1671]: time=2026-09-25T15:02:19.973Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)" lb # [ 179.429559] niks3-server[1213]: 2026/09/25 15:02:19 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst worker1 # [ 179.459278] nix-grpc-daemon[1671]: time=2026-09-25T15:02:19.991Z level=INFO msg="Uploading 1 narinfos" lb # [ 179.438429] niks3-server[1213]: 2026/09/25 15:02:19 INFO Received sign narinfos request method=POST path=/api/pushes/57/sign lb # [ 179.443966] niks3-server[1213]: 2026/09/25 15:02:19 INFO Signed narinfos id=57 count=1 lb # [ 179.447499] niks3-server[1213]: 2026/09/25 15:02:19 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls lb # [ 179.456206] niks3-server[1213]: 2026/09/25 15:02:19 INFO Received complete push request method=POST path=/api/pushes/57/complete worker1 # [ 179.486101] nix-grpc-daemon[1671]: time=2026-09-25T15:02:20.018Z level=INFO msg="Upload complete. (175ms)" lb # [ 179.462210] niks3-server[1213]: 2026/09/25 15:02:19 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo worker1 # [ 179.491297] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=768 worker1 # [ 179.500140] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 179.505841] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 179.507818] nix-daemon[855]: reaped child process 1795, status = succeeded client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051... worker1 # [ 179.516324] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) client # farm-top> node-a: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-during-drain worker1 # [ 180.752509] nix-daemon[855]: reaped child process 1804, status = succeeded worker1 # [ 180.850610] nix-daemon[855]: accepted connection from pid 1820, user nix-grpc-daemon (trusted) worker1 # [ 180.866753] nix-daemon[855]: reaped child process 1823, status = succeeded lb # [ 180.848562] niks3-server[1213]: 2026/09/25 15:02:21 INFO Received push request method=POST path=/api/pushes worker1 # [ 180.878632] nix-grpc-daemon[1671]: time=2026-09-25T15:02:21.408Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 180.880222] nix-grpc-daemon[1671]: time=2026-09-25T15:02:21.409Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)" lb # [ 180.869391] niks3-server[1213]: 2026/09/25 15:02:21 INFO Received sign narinfos request method=POST path=/api/pushes/58/sign worker1 # [ 180.898468] nix-grpc-daemon[1671]: time=2026-09-25T15:02:21.430Z level=INFO msg="Uploading 1 narinfos" lb # [ 180.876970] niks3-server[1213]: 2026/09/25 15:02:21 INFO Signed narinfos id=58 count=1 lb # [ 180.882779] niks3-server[1213]: 2026/09/25 15:02:21 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls lb # [ 180.888795] niks3-server[1213]: 2026/09/25 15:02:21 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst worker1 # [ 180.921895] nix-grpc-daemon[1671]: time=2026-09-25T15:02:21.453Z level=INFO msg="Upload complete. (196ms)" lb # [ 180.899789] niks3-server[1213]: 2026/09/25 15:02:21 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv lb # [ 180.906043] niks3-server[1213]: 2026/09/25 15:02:21 INFO Received complete push request method=POST path=/api/pushes/58/complete worker1 # [ 180.928616] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=1 lb # [ 180.909348] niks3-server[1213]: 2026/09/25 15:02:21 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo worker1 # [ 180.939163] nix-daemon[855]: reaped child process 1802, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 5.26 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.14 seconds) worker2: must succeed: pkill -f 'read -t [9]0 x' worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds) client: waiting for success: ! systemctl is-active sw worker2 # [ 181.194039] nix-daemon[845]: reaped child process 1712, status = succeeded worker2 # [ 181.284550] nix-daemon[845]: accepted connection from pid 1886, user nix-grpc-daemon (trusted) lb # [ 181.307052] niks3-server[1213]: 2026/09/25 15:02:21 INFO Received push request method=POST path=/api/pushes worker2 # [ 181.306245] nix-daemon[845]: reaped child process 1889, status = succeeded worker2 # [ 181.309392] nix-grpc-daemon[1650]: time=2026-09-25T15:02:21.850Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 181.310949] nix-grpc-daemon[1650]: time=2026-09-25T15:02:21.851Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)" lb # [ 181.329023] niks3-server[1213]: 2026/09/25 15:02:21 INFO Received sign narinfos request method=POST path=/api/pushes/59/sign worker2 # [ 181.328290] nix-grpc-daemon[1650]: time=2026-09-25T15:02:21.870Z level=INFO msg="Uploading 1 narinfos" lb # [ 181.337196] niks3-server[1213]: 2026/09/25 15:02:21 INFO Signed narinfos id=59 count=1 lb # [ 181.341409] niks3-server[1213]: 2026/09/25 15:02:21 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst lb # [ 181.352204] niks3-server[1213]: 2026/09/25 15:02:21 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv lb # [ 181.354830] niks3-server[1213]: 2026/09/25 15:02:21 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls worker2 # [ 181.356188] nix-grpc-daemon[1650]: time=2026-09-25T15:02:21.898Z level=INFO msg="Upload complete. (188ms)" lb # [ 181.361835] niks3-server[1213]: 2026/09/25 15:02:21 INFO Received complete push request method=POST path=/api/pushes/59/complete lb # [ 181.365713] niks3-server[1213]: 2026/09/25 15:02:21 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo worker2 # [ 181.364516] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59038 duration_s=13 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=1 worker2 # [ 181.374278] nix-daemon[845]: reaped child process 1710, status = succeeded client # [ 181.397475] systemd[1]: sw.service: Deactivated successfully. client # [ 181.400694] systemd[1]: sw.service: Consumed 168ms CPU time over 14.750s wall clock time, 41.4M memory peak, 7.6K incoming IP traffic, 8.5K outgoing IP traffic. worker2 # [ 181.607319] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:22Z level=info event=signal_exit client: (finished: waiting for success: ! systemctl is-active sw, in 1.07 seconds) client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; } client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.04 seconds) worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service worker2 # [ 182.271453] systemd[1]: Starting scheduler-token.service... worker2 # [ 182.327945] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 182.331799] systemd[1]: Finished scheduler-token.service. worker2 # [ 182.333226] systemd[1]: scheduler-token.service: Consumed 34ms CPU time over 59ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb # [ 182.612807] niks3-server[1213]: 2026/09/25 15:02:23 INFO lead: released remote=192.168.1.4:52130 worker2 # [ 183.608102] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:24Z level=info event=scheduler_restarting worker1 # [ 183.637258] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:24Z level=info event=scheduler_restarting addr=lb:50051 lb # [ 183.612125] envoy[897]: [2026-09-25 15:02:24.134][897][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream} worker2 # [ 183.612083] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:24Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 183.617798] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:24Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:59030 worker2 # [ 183.621039] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:24Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 183.623432] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:24Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:59030 worker2 # [ 183.626261] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:24Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 183.641981] nix-daemon[845]: reaped child process 1621, status = succeeded worker1 # [ 183.837259] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 183.840179] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 183.823213] nix-grpc-daemon[1567]: ts=2026-09-25T15:02:24Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker1 # [ 184.040574] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 184.043348] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 184.243749] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 184.246674] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 184.447083] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 184.449964] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 184.651496] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 184.654079] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 184.630970] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 184.635489] systemd[1]: nix-grpc-daemon.service: Consumed 713ms CPU time over 28.305s wall clock time, 37.8M memory peak, 235.2K incoming IP traffic, 69.1K outgoing IP traffic. worker2 # [ 184.644807] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1. worker2 # [ 184.652570] systemd[1]: Starting scheduler-token.service... worker2 # [ 184.707557] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 184.712380] systemd[1]: Finished scheduler-token.service. worker2 # [ 184.713530] systemd[1]: scheduler-token.service: Consumed 34ms CPU time over 58ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 184.726679] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 184.851894] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 184.854837] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 184.828582] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:25Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 184.830380] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:25Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 184.834472] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:25Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 184.839625] nix-grpc-daemon[1910]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 184.842735] nix-grpc-daemon[1910]: E0000 00:00:1790348545.381388 1910 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 184.856532] niks3-server[1213]: 2026/09/25 15:02:25 INFO lead: acquired remote=192.168.1.4:42344 worker2 # [ 184.864199] nix-grpc-daemon[1910]: E0000 00:00:1790348545.385121 1910 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 184.866585] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:25Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.67 seconds) lb: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 184.891496] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:25Z level=info event=scheduler_take_over worker2 # [ 184.893258] nix-grpc-daemon[1910]: E0000 00:00:1790348545.395462 1950 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 184.898813] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 184.901485] nix-grpc-daemon[1910]: E0000 00:00:1790348545.416403 1950 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 184.904319] nix-grpc-daemon[1910]: E0000 00:00:1790348545.416586 1950 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 184.906780] nix-grpc-daemon[1910]: E0000 00:00:1790348545.433109 1950 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 184.909351] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:25Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 184.910701] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1 # [ 185.055261] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 185.058040] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 185.105029] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:25Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 185.107858] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 185.258644] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 185.261393] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 185.308529] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:25Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 185.311418] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 185.461969] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 185.464768] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 185.511979] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:26Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 185.514730] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 185.665729] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:26Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 185.640862] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:26Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:48160 worker2 # [ 185.643564] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:26Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker2 # [ 185.715380] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:26Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 185.722733] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:26Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 185.725307] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:26Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:48160 worker2 # [ 185.728186] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:26Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb # [ 188.273513] envoy[897]: conn peer=[2001:db8:1::1]:41526 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9578 tx=4357 ms=5066 lb # [ 188.276479] envoy[897]: conn peer=[2001:db8:1::1]:37268 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2982 tx=1559 ms=14561 lb # [ 188.278977] envoy[897]: conn peer=[2001:db8:1::4]:41892 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=1528 tx=378 ms=27163 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2 # [ 189.362593] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:29Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:48164 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: (finished: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux, in 5.23 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 23.58 seconds) subtest: queue gauges show a burst while every build blocks lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) client: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]' client # [ 190.338059] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto --impure --expr "map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ \"q0\" \"q1\" \"q2\" \"q3\" ]". client # Running as unit: burst.service; invocation ID: 0d88e9d808d14c6589a45d2da8429adf client: (finished: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]', in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker2 # [ 190.511720] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 190.520651] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) client # [ 190.536556] nix[1502]: don't know how to build these paths: client # [ 190.537598] nix[1502]: /nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv client # [ 190.538685] nix[1502]: /nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv client # [ 190.539760] nix[1502]: /nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv client # [ 190.541774] nix[1502]: /nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker2 # [ 190.542383] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:31Z level=debug event=want client=1 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv cached=0 worker1 # [ 190.573426] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker2 # [ 190.550513] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:31Z level=debug event=assigned drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 190.583195] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker2 # [ 190.556371] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:31Z level=debug event=want client=1 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv cached=0 worker2 # [ 190.559235] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:31Z level=debug event=assigned drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv worker=192.168.1.4:50051 clients=1 client # [ 190.569565] nix[1502]: copying 1 paths to 'lb:50051'... worker2 # [ 190.562688] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:31Z level=debug event=want client=1 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv cached=0 worker1 # [ 190.591676] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 client # [ 190.570903] nix[1502]: copying path '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' to 'lb:50051'... worker2 # [ 190.565830] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:31Z level=debug event=want client=1 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv cached=0 worker2 # [ 190.568869] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 worker1 # [ 190.600580] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker2 # [ 190.571613] nix-grpc-daemon[1910]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 44 ms (attempt 1/5) worker2 # [ 190.601465] nix-grpc-daemon[1910]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 144 ms (attempt 2/5) worker1 # [ 190.729387] nix-daemon[855]: accepted connection from pid 1827, user nix-grpc-daemon (trusted) worker1 # [ 190.747913] nix-daemon[855]: reaped child process 1830, status = succeeded lb # [ 190.728962] niks3-server[1213]: 2026/09/25 15:02:31 INFO Received push request method=POST path=/api/pushes worker1 # [ 190.758617] nix-grpc-daemon[1671]: time=2026-09-25T15:02:31.289Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 190.760260] nix-grpc-daemon[1671]: time=2026-09-25T15:02:31.290Z level=INFO msg="Uploading d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv (416B)" lb # [ 190.748301] niks3-server[1213]: 2026/09/25 15:02:31 INFO Received sign narinfos request method=POST path=/api/pushes/60/sign worker2 # [ 190.746678] nix-grpc-daemon[1910]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 386 ms (attempt 3/5) lb # [ 190.757031] niks3-server[1213]: 2026/09/25 15:02:31 INFO Registered completed upload object_key=nar/06ismykzsix0w29371dm1hbzzf3ljf9jn7dyh8skp5wqfkkw0vlk.nar.zst lb # [ 190.766193] niks3-server[1213]: 2026/09/25 15:02:31 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.ls lb # [ 190.776921] niks3-server[1213]: 2026/09/25 15:02:31 INFO Signed narinfos id=60 count=1 worker1 # [ 190.800752] nix-grpc-daemon[1671]: time=2026-09-25T15:02:31.332Z level=INFO msg="Uploading 1 narinfos" lb # [ 190.786791] niks3-server[1213]: 2026/09/25 15:02:31 INFO Received complete push request method=POST path=/api/pushes/60/complete lb # [ 190.794659] niks3-server[1213]: 2026/09/25 15:02:31 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.narinfo worker1 # [ 190.815531] nix-grpc-daemon[1671]: time=2026-09-25T15:02:31.347Z level=INFO msg="Upload complete. (204ms)" worker1 # [ 190.824585] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 190.827802] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 190.838419] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 190.842757] nix-daemon[855]: reaped child process 1825, status = succeeded client # [ 190.828843] nix[1502]: building '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' on lb:50051... worker1 # [ 190.852198] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker2 # [ 191.148469] nix-grpc-daemon[1910]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 362 ms (attempt 4/5) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 191.517260] nix-grpc-daemon[1910]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 191.519369] nix-grpc-daemon[1910]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 191.532603] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48172 client # [ 191.548898] nix[1502]: copying 1 paths to 'lb:50051'... worker2 # [ 191.543918] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48172 client # [ 191.550690] nix[1502]: copying path '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' to 'lb:50051'... lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2 # [ 191.556592] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) worker2 # [ 191.591621] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:32Z level=info event=niks3_push_started pid=1984 worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.17 seconds) worker2 # [ 191.725136] nix-grpc-daemon[1984]: time=2026-09-25T15:02:32.267Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 191.838334] nix-daemon[845]: accepted connection from pid 1996, user nix-grpc-daemon (trusted) worker1 # [ 191.885438] nix-daemon[855]: reaped child process 1834, status = succeeded lb # [ 191.873909] niks3-server[1213]: 2026/09/25 15:02:32 INFO Received push request method=POST path=/api/pushes worker2 # [ 191.872108] nix-daemon[845]: reaped child process 2003, status = succeeded worker2 # [ 191.878969] nix-grpc-daemon[1984]: time=2026-09-25T15:02:32.418Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 191.881567] nix-grpc-daemon[1984]: time=2026-09-25T15:02:32.418Z level=INFO msg="Uploading dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv (416B)" lb # [ 191.899750] niks3-server[1213]: 2026/09/25 15:02:32 INFO Received sign narinfos request method=POST path=/api/pushes/61/sign worker2 # [ 191.901189] nix-grpc-daemon[1984]: time=2026-09-25T15:02:32.443Z level=INFO msg="Uploading 1 narinfos" lb # [ 191.908638] niks3-server[1213]: 2026/09/25 15:02:32 INFO Signed narinfos id=61 count=1 lb # [ 191.914554] niks3-server[1213]: 2026/09/25 15:02:32 INFO Registered completed upload object_key=nar/06mdwsnf16qmfi86y8r33asi4vcrp5z7kdxqki87psgfpgvvpdki.nar.zst lb # [ 191.924714] niks3-server[1213]: 2026/09/25 15:02:32 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.ls lb # [ 191.932339] niks3-server[1213]: 2026/09/25 15:02:32 INFO Received complete push request method=POST path=/api/pushes/61/complete worker2 # [ 191.934168] nix-grpc-daemon[1984]: time=2026-09-25T15:02:32.476Z level=INFO msg="Upload complete. (205ms)" lb # [ 191.939306] niks3-server[1213]: 2026/09/25 15:02:32 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.narinfo worker2 # [ 191.940083] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48172 duration_s=0 paths=1 nar_bytes_in=416 worker2 # [ 191.949479] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 191.953159] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) worker1 # [ 191.982195] nix-daemon[855]: accepted connection from pid 1855, user nix-grpc-daemon (trusted) worker2 # [ 191.956650] nix-daemon[845]: reaped child process 1981, status = succeeded client # [ 191.968497] nix[1502]: building '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' on lb:50051... worker2 # [ 191.965694] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) lb # [ 191.977505] niks3-server[1213]: 2026/09/25 15:02:32 INFO Received push request method=POST path=/api/pushes worker1 # [ 192.002888] nix-daemon[855]: reaped child process 1858, status = succeeded worker1 # [ 192.008309] nix-grpc-daemon[1671]: time=2026-09-25T15:02:32.537Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 192.009817] nix-grpc-daemon[1671]: time=2026-09-25T15:02:32.537Z level=INFO msg="Uploading k6zivhvzgmlyxfmy64fzd759bxlhf9b5-slow-q0 (120B)" lb # [ 191.998847] niks3-server[1213]: 2026/09/25 15:02:32 INFO Registered completed upload object_key=log/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv worker1 # [ 192.031624] nix-grpc-daemon[1671]: time=2026-09-25T15:02:32.563Z level=INFO msg="Uploading 1 narinfos" lb # [ 192.007904] niks3-server[1213]: 2026/09/25 15:02:32 INFO Received sign narinfos request method=POST path=/api/pushes/62/sign lb # [ 192.014450] niks3-server[1213]: 2026/09/25 15:02:32 INFO Signed narinfos id=62 count=1 lb # [ 192.019013] niks3-server[1213]: 2026/09/25 15:02:32 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.ls lb # [ 192.026366] niks3-server[1213]: 2026/09/25 15:02:32 INFO Registered completed upload object_key=nar/0xr809jj32p5n6mnc6i7lqsz48wrm5f9jp14m77k724041711iq7.nar.zst lb # [ 192.034975] niks3-server[1213]: 2026/09/25 15:02:32 INFO Received complete push request method=POST path=/api/pushes/62/complete worker1 # [ 192.061645] nix-grpc-daemon[1671]: time=2026-09-25T15:02:32.593Z level=INFO msg="Upload complete. (199ms)" lb # [ 192.040366] niks3-server[1213]: 2026/09/25 15:02:32 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.narinfo worker2 # [ 192.042103] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:32Z level=debug event=assigned drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 192.069015] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=1 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv assign_id=1 outputs=1 worker1 # [ 192.081581] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 192.085335] nix-daemon[855]: reaped child process 1832, status = succeeded worker1 # [ 192.089909] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 client # [ 192.075432] nix[1502]: copying 1 paths to 'lb:50051'... worker1 # [ 192.097365] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 client # [ 192.076940] nix[1502]: copying path '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' to 'lb:50051'... worker1 # [ 192.104132] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 192.218343] nix-daemon[855]: accepted connection from pid 1862, user nix-grpc-daemon (trusted) lb # [ 192.212632] niks3-server[1213]: 2026/09/25 15:02:32 INFO Received push request method=POST path=/api/pushes worker1 # [ 192.238943] nix-daemon[855]: reaped child process 1865, status = succeeded worker1 # [ 192.241302] nix-grpc-daemon[1671]: time=2026-09-25T15:02:32.772Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 192.242849] nix-grpc-daemon[1671]: time=2026-09-25T15:02:32.773Z level=INFO msg="Uploading jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv (416B)" lb # [ 192.227886] niks3-server[1213]: 2026/09/25 15:02:32 INFO Received sign narinfos request method=POST path=/api/pushes/63/sign worker1 # [ 192.256319] nix-grpc-daemon[1671]: time=2026-09-25T15:02:32.788Z level=INFO msg="Uploading 1 narinfos" lb # [ 192.235828] niks3-server[1213]: 2026/09/25 15:02:32 INFO Signed narinfos id=63 count=1 lb # [ 192.240827] niks3-server[1213]: 2026/09/25 15:02:32 INFO Registered completed upload object_key=nar/1fyfjzlx8vymsfmdv5cv8ymgjkr18fwm737qvys29plwwiwmyvwr.nar.zst lb # [ 192.250426] niks3-server[1213]: 2026/09/25 15:02:32 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.ls worker1 # [ 192.277085] nix-grpc-daemon[1671]: time=2026-09-25T15:02:32.809Z level=INFO msg="Upload complete. (164ms)" lb # [ 192.254591] niks3-server[1213]: 2026/09/25 15:02:32 INFO Received complete push request method=POST path=/api/pushes/63/complete worker1 # [ 192.281867] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=416 lb # [ 192.260284] niks3-server[1213]: 2026/09/25 15:02:32 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.narinfo worker1 # [ 192.288452] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 192.296090] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 192.299387] nix-daemon[855]: reaped child process 1860, status = succeeded client # [ 192.284269] nix[1502]: building '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' on lb:50051... worker1 # [ 192.308774] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker2 # [ 194.191872] nix-daemon[845]: reaped child process 2007, status = succeeded worker2 # [ 194.278845] nix-daemon[845]: accepted connection from pid 2032, user nix-grpc-daemon (trusted) worker2 # [ 194.297613] nix-daemon[845]: reaped child process 2035, status = succeeded lb # [ 194.307208] niks3-server[1213]: 2026/09/25 15:02:34 INFO Received push request method=POST path=/api/pushes worker2 # [ 194.312384] nix-grpc-daemon[1984]: time=2026-09-25T15:02:34.852Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 194.313936] nix-grpc-daemon[1984]: time=2026-09-25T15:02:34.852Z level=INFO msg="Uploading y3qkgrgq2zzpgvi2rf5chq94i627wkdb-slow-q3 (120B)" lb # [ 194.329574] niks3-server[1213]: 2026/09/25 15:02:34 INFO Registered completed upload object_key=nar/1v1grzkvw3yj09x0dkdm2avihvqvnri156qzp768hk85zbw7lkn9.nar.zst worker1 # [ 194.364892] nix-daemon[855]: reaped child process 1869, status = succeeded lb # [ 194.340897] niks3-server[1213]: 2026/09/25 15:02:34 INFO Registered completed upload object_key=log/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv worker2 # [ 194.341424] nix-grpc-daemon[1984]: time=2026-09-25T15:02:34.883Z level=INFO msg="Uploading 1 narinfos" lb # [ 194.347757] niks3-server[1213]: 2026/09/25 15:02:34 INFO Received sign narinfos request method=POST path=/api/pushes/64/sign lb # [ 194.353547] niks3-server[1213]: 2026/09/25 15:02:34 INFO Signed narinfos id=64 count=1 lb # [ 194.355273] niks3-server[1213]: 2026/09/25 15:02:34 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.ls lb # [ 194.364145] niks3-server[1213]: 2026/09/25 15:02:34 INFO Received complete push request method=POST path=/api/pushes/64/complete lb # [ 194.371572] niks3-server[1213]: 2026/09/25 15:02:34 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.narinfo worker2 # [ 194.372596] nix-grpc-daemon[1984]: time=2026-09-25T15:02:34.915Z level=INFO msg="Upload complete. (197ms)" worker2 # [ 194.384599] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 duration_s=2 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv assign_id=2 outputs=1 worker2 # [ 194.389728] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:34Z level=debug event=assigned drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 194.398812] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 194.402850] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 194.406677] nix-daemon[845]: reaped child process 2005, status = succeeded client # [ 194.414921] nix[1502]: copying 1 paths to 'lb:50051'... worker2 # [ 194.409248] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48172 client # [ 194.415076] nix[1502]: copying path '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' to 'lb:50051'... worker2 # [ 194.418536] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) worker1 # [ 194.462168] nix-daemon[855]: accepted connection from pid 1894, user nix-grpc-daemon (trusted) lb # [ 194.455678] niks3-server[1213]: 2026/09/25 15:02:34 INFO Received push request method=POST path=/api/pushes worker1 # [ 194.481750] nix-daemon[855]: reaped child process 1897, status = succeeded worker1 # [ 194.484406] nix-grpc-daemon[1671]: time=2026-09-25T15:02:35.015Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 194.485895] nix-grpc-daemon[1671]: time=2026-09-25T15:02:35.016Z level=INFO msg="Uploading gjyi8j90pk5flrsqqdhj6h132pcb7n33-slow-q2 (120B)" lb # [ 194.483669] niks3-server[1213]: 2026/09/25 15:02:35 INFO Received sign narinfos request method=POST path=/api/pushes/65/sign worker1 # [ 194.511595] nix-grpc-daemon[1671]: time=2026-09-25T15:02:35.043Z level=INFO msg="Uploading 1 narinfos" lb # [ 194.492560] niks3-server[1213]: 2026/09/25 15:02:35 INFO Signed narinfos id=65 count=1 lb # [ 194.496734] niks3-server[1213]: 2026/09/25 15:02:35 INFO Registered completed upload object_key=nar/15imdx7zyy6mzjjv6qvdzywh0d9w1n6ikc0rq4rq95aw9x6swfzj.nar.zst lb # [ 194.502897] niks3-server[1213]: 2026/09/25 15:02:35 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.ls lb # [ 194.510156] niks3-server[1213]: 2026/09/25 15:02:35 INFO Registered completed upload object_key=log/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv lb # [ 194.515675] niks3-server[1213]: 2026/09/25 15:02:35 INFO Received complete push request method=POST path=/api/pushes/65/complete worker1 # [ 194.543364] nix-grpc-daemon[1671]: time=2026-09-25T15:02:35.075Z level=INFO msg="Upload complete. (226ms)" lb # [ 194.520656] niks3-server[1213]: 2026/09/25 15:02:35 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.narinfo worker2 # [ 194.527491] nix-daemon[845]: accepted connection from pid 2039, user nix-grpc-daemon (trusted) worker1 # [ 194.552642] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=2 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv assign_id=3 outputs=1 worker1 # [ 194.560941] nix-daemon[855]: reaped child process 1867, status = succeeded lb # [ 194.549602] niks3-server[1213]: 2026/09/25 15:02:35 INFO Received push request method=POST path=/api/pushes worker2 # [ 194.548859] nix-daemon[845]: reaped child process 2042, status = succeeded worker2 # [ 194.550538] nix-grpc-daemon[1984]: time=2026-09-25T15:02:35.092Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 194.552256] nix-grpc-daemon[1984]: time=2026-09-25T15:02:35.092Z level=INFO msg="Uploading pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv (416B)" lb # [ 194.564848] niks3-server[1213]: 2026/09/25 15:02:35 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.ls worker2 # [ 194.568390] nix-grpc-daemon[1984]: time=2026-09-25T15:02:35.110Z level=INFO msg="Uploading 1 narinfos" lb # [ 194.572195] niks3-server[1213]: 2026/09/25 15:02:35 INFO Received sign narinfos request method=POST path=/api/pushes/66/sign lb # [ 194.575707] niks3-server[1213]: 2026/09/25 15:02:35 INFO Signed narinfos id=66 count=1 lb # [ 194.579442] niks3-server[1213]: 2026/09/25 15:02:35 INFO Registered completed upload object_key=nar/1k21jrz411xc9gkca2pnr9mrjq1s3bg3zhsam3nnv4j066dgzgm7.nar.zst lb # [ 194.589910] niks3-server[1213]: 2026/09/25 15:02:35 INFO Received complete push request method=POST path=/api/pushes/66/complete worker2 # [ 194.590829] nix-grpc-daemon[1984]: time=2026-09-25T15:02:35.133Z level=INFO msg="Upload complete. (165ms)" lb # [ 194.595571] niks3-server[1213]: 2026/09/25 15:02:35 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.narinfo worker2 # [ 194.596176] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48172 duration_s=0 paths=1 nar_bytes_in=416 worker2 # [ 194.605374] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 194.609649] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) worker2 # [ 194.613575] nix-daemon[845]: reaped child process 2037, status = succeeded client # [ 194.625787] nix[1502]: building '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' on lb:50051... worker2 # [ 194.623104] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) worker2 # [ 196.470073] nix-daemon[845]: reaped child process 2046, status = succeeded worker2 # [ 196.535867] nix-daemon[845]: accepted connection from pid 2071, user nix-grpc-daemon (trusted) worker2 # [ 196.551249] nix-daemon[845]: reaped child process 2074, status = succeeded lb # [ 196.561103] niks3-server[1213]: 2026/09/25 15:02:37 INFO Received push request method=POST path=/api/pushes worker2 # [ 196.563905] nix-grpc-daemon[1984]: time=2026-09-25T15:02:37.105Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 196.565631] nix-grpc-daemon[1984]: time=2026-09-25T15:02:37.105Z level=INFO msg="Uploading l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx-slow-q1 (120B)" lb # [ 196.587159] niks3-server[1213]: 2026/09/25 15:02:37 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.ls worker2 # [ 196.590805] nix-grpc-daemon[1984]: time=2026-09-25T15:02:37.133Z level=INFO msg="Uploading 1 narinfos" lb # [ 196.596315] niks3-server[1213]: 2026/09/25 15:02:37 INFO Received sign narinfos request method=POST path=/api/pushes/67/sign lb # [ 196.598590] niks3-server[1213]: 2026/09/25 15:02:37 INFO Registered completed upload object_key=nar/0c1c367r3msa4fz6imwp70zxi4fxvrkb86v3hv921wsrf13yd8v3.nar.zst lb # [ 196.607462] niks3-server[1213]: 2026/09/25 15:02:37 INFO Signed narinfos id=67 count=1 lb # [ 196.614927] niks3-server[1213]: 2026/09/25 15:02:37 INFO Registered completed upload object_key=log/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker2 # [ 196.614317] nix-grpc-daemon[1984]: time=2026-09-25T15:02:37.156Z level=INFO msg="Upload complete. (183ms)" lb # [ 196.619190] niks3-server[1213]: 2026/09/25 15:02:37 INFO Received complete push request method=POST path=/api/pushes/67/complete lb # [ 196.623865] niks3-server[1213]: 2026/09/25 15:02:37 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.narinfo worker2 # [ 196.624626] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 duration_s=2 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv assign_id=4 outputs=1 worker2 # [ 196.632584] nix-daemon[845]: reaped child process 2044, status = succeeded client # [ 196.652257] systemd[1]: burst.service: Deactivated successfully. client # [ 196.654544] systemd[1]: burst.service: Consumed 191ms CPU time over 6.316s wall clock time, 53.6M memory peak, 14K incoming IP traffic, 19.2K outgoing IP traffic. client: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; } client: (finished: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: queue gauges show a burst while every build blocks, in 7.51 seconds) subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1 worker2 # [ 197.873293] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:38Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:48172 worker1 # [ 197.904006] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker2 # [ 197.901034] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:38Z level=debug event=want client=2 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=0 worker2 # [ 197.903903] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:38Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=1 worker2 # [ 197.907494] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:38Z level=debug event=want client=2 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=0 worker1 # [ 197.937150] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker2 # [ 197.910629] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:38Z level=debug event=want client=2 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=0 worker1 # [ 197.945768] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 197.953917] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 197.963035] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 198.094928] nix-daemon[855]: accepted connection from pid 1913, user nix-grpc-daemon (trusted) worker1 # [ 198.113061] nix-daemon[855]: reaped child process 1916, status = succeeded lb # [ 198.092753] niks3-server[1213]: 2026/09/25 15:02:38 INFO Received push request method=POST path=/api/pushes worker1 # [ 198.121968] nix-grpc-daemon[1671]: time=2026-09-25T15:02:38.653Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 198.123700] nix-grpc-daemon[1671]: time=2026-09-25T15:02:38.653Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)" lb # [ 198.108108] niks3-server[1213]: 2026/09/25 15:02:38 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst worker1 # [ 198.141199] nix-grpc-daemon[1671]: time=2026-09-25T15:02:38.673Z level=INFO msg="Uploading 1 narinfos" lb # [ 198.114299] niks3-server[1213]: 2026/09/25 15:02:38 INFO Received sign narinfos request method=POST path=/api/pushes/68/sign lb # [ 198.120514] niks3-server[1213]: 2026/09/25 15:02:38 INFO Signed narinfos id=68 count=1 lb # [ 198.124692] niks3-server[1213]: 2026/09/25 15:02:38 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls lb # [ 198.132464] niks3-server[1213]: 2026/09/25 15:02:38 INFO Received complete push request method=POST path=/api/pushes/68/complete worker1 # [ 198.163570] nix-grpc-daemon[1671]: time=2026-09-25T15:02:38.695Z level=INFO msg="Upload complete. (190ms)" lb # [ 198.140095] niks3-server[1213]: 2026/09/25 15:02:38 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo worker1 # [ 198.169598] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 198.176719] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 198.185061] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 198.189020] nix-daemon[855]: reaped child process 1911, status = succeeded worker1 # [ 198.199254] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) lb # [ 198.274092] envoy[897]: conn peer=[2001:db8:1::1]:44104 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9714 tx=5240 ms=6125 worker1 # [ 199.652630] nix-daemon[855]: reaped child process 1920, status = succeeded worker1 # [ 199.742095] nix-daemon[855]: accepted connection from pid 1936, user nix-grpc-daemon (trusted) worker1 # [ 199.758007] nix-daemon[855]: reaped child process 1939, status = succeeded lb # [ 199.741279] niks3-server[1213]: 2026/09/25 15:02:40 INFO Received push request method=POST path=/api/pushes worker1 # [ 199.772379] nix-grpc-daemon[1671]: time=2026-09-25T15:02:40.303Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 199.773930] nix-grpc-daemon[1671]: time=2026-09-25T15:02:40.303Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)" lb # [ 199.770551] niks3-server[1213]: 2026/09/25 15:02:40 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker1 # [ 199.800654] nix-grpc-daemon[1671]: time=2026-09-25T15:02:40.332Z level=INFO msg="Uploading 1 narinfos" lb # [ 199.779549] niks3-server[1213]: 2026/09/25 15:02:40 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls lb # [ 199.785448] niks3-server[1213]: 2026/09/25 15:02:40 INFO Received sign narinfos request method=POST path=/api/pushes/69/sign lb # [ 199.787805] niks3-server[1213]: 2026/09/25 15:02:40 INFO Signed narinfos id=69 count=1 lb # [ 199.797282] niks3-server[1213]: 2026/09/25 15:02:40 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst worker1 # [ 199.826218] nix-grpc-daemon[1671]: time=2026-09-25T15:02:40.358Z level=INFO msg="Upload complete. (218ms)" lb # [ 199.803991] niks3-server[1213]: 2026/09/25 15:02:40 INFO Received complete push request method=POST path=/api/pushes/69/complete lb # [ 199.807474] niks3-server[1213]: 2026/09/25 15:02:40 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo worker2 # [ 199.804775] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:40Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 199.832647] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=5 outputs=1 worker2 # [ 199.812081] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:40Z level=debug event=want client=2 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=0 worker1 # [ 199.844948] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 199.848831] nix-daemon[855]: reaped child process 1918, status = succeeded worker1 # [ 199.853801] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 199.861742] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 199.869559] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 199.989453] nix-daemon[855]: accepted connection from pid 1943, user nix-grpc-daemon (trusted) lb # [ 199.984666] niks3-server[1213]: 2026/09/25 15:02:40 INFO Received push request method=POST path=/api/pushes worker1 # [ 200.010645] nix-daemon[855]: reaped child process 1946, status = succeeded worker1 # [ 200.013305] nix-grpc-daemon[1671]: time=2026-09-25T15:02:40.545Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 200.014977] nix-grpc-daemon[1671]: time=2026-09-25T15:02:40.545Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)" lb # [ 200.000633] niks3-server[1213]: 2026/09/25 15:02:40 INFO Received sign narinfos request method=POST path=/api/pushes/70/sign worker1 # [ 200.029274] nix-grpc-daemon[1671]: time=2026-09-25T15:02:40.561Z level=INFO msg="Uploading 1 narinfos" lb # [ 200.008603] niks3-server[1213]: 2026/09/25 15:02:40 INFO Signed narinfos id=70 count=1 lb # [ 200.010363] niks3-server[1213]: 2026/09/25 15:02:40 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls lb # [ 200.020381] niks3-server[1213]: 2026/09/25 15:02:40 INFO Received complete push request method=POST path=/api/pushes/70/complete worker1 # [ 200.050027] nix-grpc-daemon[1671]: time=2026-09-25T15:02:40.582Z level=INFO msg="Upload complete. (171ms)" lb # [ 200.026981] niks3-server[1213]: 2026/09/25 15:02:40 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst worker1 # [ 200.054940] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 200.031898] niks3-server[1213]: 2026/09/25 15:02:40 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo worker1 # [ 200.063673] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 200.067432] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 200.070811] nix-daemon[855]: reaped child process 1941, status = succeeded worker1 # [ 200.078626] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 201.153658] nix-daemon[855]: reaped child process 1950, status = succeeded worker1 # [ 201.218095] nix-daemon[855]: accepted connection from pid 1966, user nix-grpc-daemon (trusted) worker1 # [ 201.234231] nix-daemon[855]: reaped child process 1969, status = succeeded lb # [ 201.217068] niks3-server[1213]: 2026/09/25 15:02:41 INFO Received push request method=POST path=/api/pushes worker1 # [ 201.248383] nix-grpc-daemon[1671]: time=2026-09-25T15:02:41.777Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 201.249912] nix-grpc-daemon[1671]: time=2026-09-25T15:02:41.777Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)" lb # [ 201.242124] niks3-server[1213]: 2026/09/25 15:02:41 INFO Received sign narinfos request method=POST path=/api/pushes/71/sign worker1 # [ 201.270197] nix-grpc-daemon[1671]: time=2026-09-25T15:02:41.802Z level=INFO msg="Uploading 1 narinfos" lb # [ 201.248288] niks3-server[1213]: 2026/09/25 15:02:41 INFO Signed narinfos id=71 count=1 lb # [ 201.254406] niks3-server[1213]: 2026/09/25 15:02:41 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls lb # [ 201.260976] niks3-server[1213]: 2026/09/25 15:02:41 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv lb # [ 201.271247] niks3-server[1213]: 2026/09/25 15:02:41 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst worker1 # [ 201.301270] nix-grpc-daemon[1671]: time=2026-09-25T15:02:41.833Z level=INFO msg="Upload complete. (214ms)" lb # [ 201.278004] niks3-server[1213]: 2026/09/25 15:02:41 INFO Received complete push request method=POST path=/api/pushes/71/complete lb # [ 201.280846] niks3-server[1213]: 2026/09/25 15:02:41 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo worker1 # [ 201.308555] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=6 outputs=1 worker2 # [ 201.282712] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:41Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=1 worker2 # [ 201.289852] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:41Z level=debug event=want client=2 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=0 worker1 # [ 201.320250] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 201.323939] nix-daemon[855]: reaped child process 1948, status = succeeded worker1 # [ 201.331254] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 201.339999] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 201.347618] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 201.468824] nix-daemon[855]: accepted connection from pid 1973, user nix-grpc-daemon (trusted) worker1 # [ 201.485456] nix-daemon[855]: reaped child process 1976, status = succeeded lb # [ 201.464945] niks3-server[1213]: 2026/09/25 15:02:41 INFO Received push request method=POST path=/api/pushes worker1 # [ 201.493510] nix-grpc-daemon[1671]: time=2026-09-25T15:02:42.025Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 201.495075] nix-grpc-daemon[1671]: time=2026-09-25T15:02:42.025Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)" worker1 # [ 201.510635] nix-grpc-daemon[1671]: time=2026-09-25T15:02:42.042Z level=INFO msg="Uploading 1 narinfos" lb # [ 201.482225] niks3-server[1213]: 2026/09/25 15:02:42 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls lb # [ 201.489156] niks3-server[1213]: 2026/09/25 15:02:42 INFO Received sign narinfos request method=POST path=/api/pushes/72/sign lb # [ 201.491481] niks3-server[1213]: 2026/09/25 15:02:42 INFO Signed narinfos id=72 count=1 lb # [ 201.498977] niks3-server[1213]: 2026/09/25 15:02:42 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst lb # [ 201.509713] niks3-server[1213]: 2026/09/25 15:02:42 INFO Received complete push request method=POST path=/api/pushes/72/complete worker1 # [ 201.536764] nix-grpc-daemon[1671]: time=2026-09-25T15:02:42.068Z level=INFO msg="Upload complete. (179ms)" lb # [ 201.514099] niks3-server[1213]: 2026/09/25 15:02:42 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo worker1 # [ 201.542736] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 201.551779] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 201.557832] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 201.560227] nix-daemon[855]: reaped child process 1971, status = succeeded worker1 # [ 201.570136] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 202.212287] nix-daemon[855]: reaped child process 1980, status = succeeded worker1 # [ 202.297589] nix-daemon[855]: accepted connection from pid 1996, user nix-grpc-daemon (trusted) worker1 # [ 202.317451] nix-daemon[855]: reaped child process 1999, status = succeeded lb # [ 202.293087] niks3-server[1213]: 2026/09/25 15:02:42 INFO Received push request method=POST path=/api/pushes worker1 # [ 202.323152] nix-grpc-daemon[1671]: time=2026-09-25T15:02:42.854Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 202.324884] nix-grpc-daemon[1671]: time=2026-09-25T15:02:42.854Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)" lb # [ 202.315524] niks3-server[1213]: 2026/09/25 15:02:42 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls worker1 # [ 202.345437] nix-grpc-daemon[1671]: time=2026-09-25T15:02:42.877Z level=INFO msg="Uploading 1 narinfos" lb # [ 202.322322] niks3-server[1213]: 2026/09/25 15:02:42 INFO Received sign narinfos request method=POST path=/api/pushes/73/sign lb # [ 202.328498] niks3-server[1213]: 2026/09/25 15:02:42 INFO Signed narinfos id=73 count=1 lb # [ 202.332980] niks3-server[1213]: 2026/09/25 15:02:42 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst lb # [ 202.340264] niks3-server[1213]: 2026/09/25 15:02:42 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker1 # [ 202.372803] nix-grpc-daemon[1671]: time=2026-09-25T15:02:42.904Z level=INFO msg="Upload complete. (195ms)" lb # [ 202.349741] niks3-server[1213]: 2026/09/25 15:02:42 INFO Received complete push request method=POST path=/api/pushes/73/complete lb # [ 202.356605] niks3-server[1213]: 2026/09/25 15:02:42 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo worker2 # [ 202.352557] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:42Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 202.379443] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=7 outputs=1 worker2 # [ 202.358032] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:42Z level=debug event=want client=2 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=0 worker1 # [ 202.391737] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 202.395932] nix-daemon[855]: reaped child process 1978, status = succeeded worker1 # [ 202.401143] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 202.409419] nix-grpc-daemon[1671]: time=2026-09-25T15:02:42.941Z level=INFO msg="All 1 paths already cached" worker1 # [ 202.414752] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 202.422307] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 202.533554] nix-daemon[855]: accepted connection from pid 2003, user nix-grpc-daemon (trusted) worker1 # [ 202.551731] nix-daemon[855]: reaped child process 2006, status = succeeded lb # [ 202.532512] niks3-server[1213]: 2026/09/25 15:02:43 INFO Received push request method=POST path=/api/pushes worker1 # [ 202.561002] nix-grpc-daemon[1671]: time=2026-09-25T15:02:43.092Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 202.562564] nix-grpc-daemon[1671]: time=2026-09-25T15:02:43.092Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)" lb # [ 202.547523] niks3-server[1213]: 2026/09/25 15:02:43 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls worker1 # [ 202.578250] nix-grpc-daemon[1671]: time=2026-09-25T15:02:43.110Z level=INFO msg="Uploading 1 narinfos" lb # [ 202.554603] niks3-server[1213]: 2026/09/25 15:02:43 INFO Received sign narinfos request method=POST path=/api/pushes/74/sign lb # [ 202.559669] niks3-server[1213]: 2026/09/25 15:02:43 INFO Signed narinfos id=74 count=1 lb # [ 202.568646] niks3-server[1213]: 2026/09/25 15:02:43 INFO Received complete push request method=POST path=/api/pushes/74/complete lb # [ 202.577561] niks3-server[1213]: 2026/09/25 15:02:43 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst worker1 # [ 202.605561] nix-grpc-daemon[1671]: time=2026-09-25T15:02:43.137Z level=INFO msg="Upload complete. (174ms)" lb # [ 202.582383] niks3-server[1213]: 2026/09/25 15:02:43 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo worker1 # [ 202.611331] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 202.621131] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 202.626119] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 202.630605] nix-daemon[855]: reaped child process 2001, status = succeeded worker1 # [ 202.639960] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 203.497042] nix-daemon[855]: reaped child process 2010, status = succeeded worker1 # [ 203.566190] nix-daemon[855]: accepted connection from pid 2026, user nix-grpc-daemon (trusted) worker1 # [ 203.581661] nix-daemon[855]: reaped child process 2029, status = succeeded lb # [ 203.563704] niks3-server[1213]: 2026/09/25 15:02:44 INFO Received push request method=POST path=/api/pushes worker1 # [ 203.593944] nix-grpc-daemon[1671]: time=2026-09-25T15:02:44.124Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 203.595531] nix-grpc-daemon[1671]: time=2026-09-25T15:02:44.124Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)" lb # [ 203.587957] niks3-server[1213]: 2026/09/25 15:02:44 INFO Received sign narinfos request method=POST path=/api/pushes/75/sign worker1 # [ 203.617241] nix-grpc-daemon[1671]: time=2026-09-25T15:02:44.149Z level=INFO msg="Uploading 1 narinfos" lb # [ 203.596185] niks3-server[1213]: 2026/09/25 15:02:44 INFO Signed narinfos id=75 count=1 lb # [ 203.597904] niks3-server[1213]: 2026/09/25 15:02:44 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv lb # [ 203.607230] niks3-server[1213]: 2026/09/25 15:02:44 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls lb # [ 203.612954] niks3-server[1213]: 2026/09/25 15:02:44 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst worker1 # [ 203.644965] nix-grpc-daemon[1671]: time=2026-09-25T15:02:44.176Z level=INFO msg="Upload complete. (197ms)" lb # [ 203.621738] niks3-server[1213]: 2026/09/25 15:02:44 INFO Received complete push request method=POST path=/api/pushes/75/complete worker2 # [ 203.623296] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:44Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=1 lb # [ 203.629384] niks3-server[1213]: 2026/09/25 15:02:44 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo worker1 # [ 203.653311] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=8 outputs=1 worker2 # [ 203.630825] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:44Z level=debug event=want client=2 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=0 worker1 # [ 203.663549] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 203.667615] nix-daemon[855]: reaped child process 2008, status = succeeded worker1 # [ 203.672943] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 203.681290] nix-grpc-daemon[1671]: time=2026-09-25T15:02:44.213Z level=INFO msg="All 1 paths already cached" worker1 # [ 203.687011] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 203.694721] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 203.809718] nix-daemon[855]: accepted connection from pid 2033, user nix-grpc-daemon (trusted) worker1 # [ 203.828541] nix-daemon[855]: reaped child process 2036, status = succeeded lb # [ 203.808442] niks3-server[1213]: 2026/09/25 15:02:44 INFO Received push request method=POST path=/api/pushes worker1 # [ 203.836899] nix-grpc-daemon[1671]: time=2026-09-25T15:02:44.368Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 203.838477] nix-grpc-daemon[1671]: time=2026-09-25T15:02:44.368Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)" lb # [ 203.823071] niks3-server[1213]: 2026/09/25 15:02:44 INFO Received sign narinfos request method=POST path=/api/pushes/76/sign worker1 # [ 203.849441] nix-grpc-daemon[1671]: time=2026-09-25T15:02:44.381Z level=INFO msg="Uploading 1 narinfos" lb # [ 203.828641] niks3-server[1213]: 2026/09/25 15:02:44 INFO Signed narinfos id=76 count=1 lb # [ 203.834258] niks3-server[1213]: 2026/09/25 15:02:44 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls lb # [ 203.840708] niks3-server[1213]: 2026/09/25 15:02:44 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst lb # [ 203.848863] niks3-server[1213]: 2026/09/25 15:02:44 INFO Received complete push request method=POST path=/api/pushes/76/complete worker1 # [ 203.877189] nix-grpc-daemon[1671]: time=2026-09-25T15:02:44.409Z level=INFO msg="Upload complete. (173ms)" lb # [ 203.854615] niks3-server[1213]: 2026/09/25 15:02:44 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo worker1 # [ 203.883479] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 203.892866] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 203.897214] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 203.901127] nix-daemon[855]: reaped child process 2031, status = succeeded worker1 # [ 203.910669] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 204.943619] nix-daemon[855]: reaped child process 2040, status = succeeded worker1 # [ 205.053019] nix-daemon[855]: accepted connection from pid 2056, user nix-grpc-daemon (trusted) worker1 # [ 205.067884] nix-daemon[855]: reaped child process 2059, status = succeeded lb # [ 205.050243] niks3-server[1213]: 2026/09/25 15:02:45 INFO Received push request method=POST path=/api/pushes worker1 # [ 205.080436] nix-grpc-daemon[1671]: time=2026-09-25T15:02:45.611Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 205.082427] nix-grpc-daemon[1671]: time=2026-09-25T15:02:45.611Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)" lb # [ 205.073536] niks3-server[1213]: 2026/09/25 15:02:45 INFO Received sign narinfos request method=POST path=/api/pushes/77/sign worker1 # [ 205.101434] nix-grpc-daemon[1671]: time=2026-09-25T15:02:45.633Z level=INFO msg="Uploading 1 narinfos" lb # [ 205.080505] niks3-server[1213]: 2026/09/25 15:02:45 INFO Signed narinfos id=77 count=1 lb # [ 205.085590] niks3-server[1213]: 2026/09/25 15:02:45 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv lb # [ 205.092200] niks3-server[1213]: 2026/09/25 15:02:45 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst lb # [ 205.102108] niks3-server[1213]: 2026/09/25 15:02:45 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls worker1 # [ 205.130551] nix-grpc-daemon[1671]: time=2026-09-25T15:02:45.662Z level=INFO msg="Upload complete. (213ms)" lb # [ 205.107753] niks3-server[1213]: 2026/09/25 15:02:45 INFO Received complete push request method=POST path=/api/pushes/77/complete lb # [ 205.111438] niks3-server[1213]: 2026/09/25 15:02:45 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo worker2 # [ 205.108729] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:45Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 205.140263] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=9 outputs=1 worker2 # [ 205.117029] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:45Z level=debug event=want client=2 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=0 worker1 # [ 205.149612] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 205.153598] nix-daemon[855]: reaped child process 2038, status = succeeded worker1 # [ 205.159052] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 205.166880] nix-grpc-daemon[1671]: time=2026-09-25T15:02:45.698Z level=INFO msg="All 1 paths already cached" worker1 # [ 205.172589] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 205.179923] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 205.297919] nix-daemon[855]: accepted connection from pid 2063, user nix-grpc-daemon (trusted) lb # [ 205.290389] niks3-server[1213]: 2026/09/25 15:02:45 INFO Received push request method=POST path=/api/pushes worker1 # [ 205.317643] nix-daemon[855]: reaped child process 2066, status = succeeded worker1 # [ 205.319537] nix-grpc-daemon[1671]: time=2026-09-25T15:02:45.851Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 205.321121] nix-grpc-daemon[1671]: time=2026-09-25T15:02:45.851Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)" lb # [ 205.305867] niks3-server[1213]: 2026/09/25 15:02:45 INFO Received sign narinfos request method=POST path=/api/pushes/78/sign worker1 # [ 205.333177] nix-grpc-daemon[1671]: time=2026-09-25T15:02:45.865Z level=INFO msg="Uploading 1 narinfos" lb # [ 205.312946] niks3-server[1213]: 2026/09/25 15:02:45 INFO Signed narinfos id=78 count=1 lb # [ 205.314670] niks3-server[1213]: 2026/09/25 15:02:45 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls lb # [ 205.324461] niks3-server[1213]: 2026/09/25 15:02:45 INFO Received complete push request method=POST path=/api/pushes/78/complete lb # [ 205.329179] niks3-server[1213]: 2026/09/25 15:02:45 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst worker1 # [ 205.356955] nix-grpc-daemon[1671]: time=2026-09-25T15:02:45.888Z level=INFO msg="Upload complete. (168ms)" lb # [ 205.334151] niks3-server[1213]: 2026/09/25 15:02:45 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo worker1 # [ 205.362997] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 205.370410] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 205.375058] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 205.378534] nix-daemon[855]: reaped child process 2061, status = succeeded worker1 # [ 205.387838] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 205.845856] nix-daemon[855]: reaped child process 2070, status = succeeded worker1 # [ 205.904983] nix-daemon[855]: accepted connection from pid 2086, user nix-grpc-daemon (trusted) worker1 # [ 205.920907] nix-daemon[855]: reaped child process 2089, status = succeeded lb # [ 205.901430] niks3-server[1213]: 2026/09/25 15:02:46 INFO Received push request method=POST path=/api/pushes worker1 # [ 205.932147] nix-grpc-daemon[1671]: time=2026-09-25T15:02:46.461Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 205.933678] nix-grpc-daemon[1671]: time=2026-09-25T15:02:46.461Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)" lb # [ 205.928839] niks3-server[1213]: 2026/09/25 15:02:46 INFO Received sign narinfos request method=POST path=/api/pushes/79/sign worker1 # [ 205.957605] nix-grpc-daemon[1671]: time=2026-09-25T15:02:46.489Z level=INFO msg="Uploading 1 narinfos" lb # [ 205.937934] niks3-server[1213]: 2026/09/25 15:02:46 INFO Signed narinfos id=79 count=1 lb # [ 205.942132] niks3-server[1213]: 2026/09/25 15:02:46 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls lb # [ 205.951167] niks3-server[1213]: 2026/09/25 15:02:46 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker1 # [ 205.978980] nix-grpc-daemon[1671]: time=2026-09-25T15:02:46.511Z level=INFO msg="Upload complete. (185ms)" lb # [ 205.957581] niks3-server[1213]: 2026/09/25 15:02:46 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst lb # [ 205.961890] niks3-server[1213]: 2026/09/25 15:02:46 INFO Received complete push request method=POST path=/api/pushes/79/complete worker2 # [ 205.958212] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:46Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=1 lb # [ 205.966440] niks3-server[1213]: 2026/09/25 15:02:46 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo worker1 # [ 205.984303] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=10 outputs=1 worker2 # [ 205.967559] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:46Z level=debug event=want client=2 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=0 worker1 # [ 205.996250] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 206.001065] nix-daemon[855]: reaped child process 2068, status = succeeded worker1 # [ 206.006170] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 206.013728] nix-grpc-daemon[1671]: time=2026-09-25T15:02:46.545Z level=INFO msg="All 2 paths already cached" worker1 # [ 206.019429] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 206.026962] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 206.135962] nix-daemon[855]: accepted connection from pid 2093, user nix-grpc-daemon (trusted) worker1 # [ 206.150574] nix-daemon[855]: reaped child process 2096, status = succeeded lb # [ 206.133196] niks3-server[1213]: 2026/09/25 15:02:46 INFO Received push request method=POST path=/api/pushes worker1 # [ 206.161730] nix-grpc-daemon[1671]: time=2026-09-25T15:02:46.693Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 206.163272] nix-grpc-daemon[1671]: time=2026-09-25T15:02:46.693Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)" lb # [ 206.149052] niks3-server[1213]: 2026/09/25 15:02:46 INFO Received sign narinfos request method=POST path=/api/pushes/80/sign worker1 # [ 206.177346] nix-grpc-daemon[1671]: time=2026-09-25T15:02:46.709Z level=INFO msg="Uploading 1 narinfos" lb # [ 206.156946] niks3-server[1213]: 2026/09/25 15:02:46 INFO Signed narinfos id=80 count=1 lb # [ 206.161322] niks3-server[1213]: 2026/09/25 15:02:46 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls lb # [ 206.163754] niks3-server[1213]: 2026/09/25 15:02:46 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst lb # [ 206.174591] niks3-server[1213]: 2026/09/25 15:02:46 INFO Received complete push request method=POST path=/api/pushes/80/complete worker1 # [ 206.202662] nix-grpc-daemon[1671]: time=2026-09-25T15:02:46.734Z level=INFO msg="Upload complete. (167ms)" lb # [ 206.179115] niks3-server[1213]: 2026/09/25 15:02:46 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo worker1 # [ 206.208943] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 206.218203] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 206.224225] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 206.226709] nix-daemon[855]: reaped child process 2091, status = succeeded worker1 # [ 206.236401] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 207.465673] nix-daemon[855]: reaped child process 2100, status = succeeded worker1 # [ 207.579812] nix-daemon[855]: accepted connection from pid 2116, user nix-grpc-daemon (trusted) worker1 # [ 207.600687] nix-daemon[855]: reaped child process 2119, status = succeeded lb # [ 207.581452] niks3-server[1213]: 2026/09/25 15:02:48 INFO Received push request method=POST path=/api/pushes worker1 # [ 207.612005] nix-grpc-daemon[1671]: time=2026-09-25T15:02:48.142Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 207.613935] nix-grpc-daemon[1671]: time=2026-09-25T15:02:48.142Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)" lb # [ 207.606919] niks3-server[1213]: 2026/09/25 15:02:48 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker1 # [ 207.638178] nix-grpc-daemon[1671]: time=2026-09-25T15:02:48.170Z level=INFO msg="Uploading 1 narinfos" lb # [ 207.615609] niks3-server[1213]: 2026/09/25 15:02:48 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst lb # [ 207.622398] niks3-server[1213]: 2026/09/25 15:02:48 INFO Received sign narinfos request method=POST path=/api/pushes/81/sign lb # [ 207.630960] niks3-server[1213]: 2026/09/25 15:02:48 INFO Signed narinfos id=81 count=1 worker1 # [ 207.658422] nix-grpc-daemon[1671]: time=2026-09-25T15:02:48.190Z level=INFO msg="Upload complete. (214ms)" lb # [ 207.635062] niks3-server[1213]: 2026/09/25 15:02:48 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls worker2 # [ 207.637125] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:48Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=1 lb # [ 207.642287] niks3-server[1213]: 2026/09/25 15:02:48 INFO Received complete push request method=POST path=/api/pushes/81/complete worker1 # [ 207.665320] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=11 outputs=1 lb # [ 207.645136] niks3-server[1213]: 2026/09/25 15:02:48 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo worker1 # [ 207.678208] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 207.682251] nix-daemon[855]: reaped child process 2098, status = succeeded worker1 # [ 207.687430] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 207.695161] nix-grpc-daemon[1671]: time=2026-09-25T15:02:48.227Z level=INFO msg="All 2 paths already cached" worker1 # [ 207.700792] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 207.709102] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 207.822113] nix-daemon[855]: accepted connection from pid 2123, user nix-grpc-daemon (trusted) lb # [ 207.818819] niks3-server[1213]: 2026/09/25 15:02:48 INFO Received push request method=POST path=/api/pushes worker1 # [ 207.844609] nix-daemon[855]: reaped child process 2126, status = succeeded worker1 # [ 207.848247] nix-grpc-daemon[1671]: time=2026-09-25T15:02:48.380Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 207.849773] nix-grpc-daemon[1671]: time=2026-09-25T15:02:48.380Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)" lb # [ 207.839006] niks3-server[1213]: 2026/09/25 15:02:48 INFO Received sign narinfos request method=POST path=/api/pushes/82/sign worker1 # [ 207.866622] nix-grpc-daemon[1671]: time=2026-09-25T15:02:48.398Z level=INFO msg="Uploading 1 narinfos" lb # [ 207.847006] niks3-server[1213]: 2026/09/25 15:02:48 INFO Signed narinfos id=82 count=1 lb # [ 207.851406] niks3-server[1213]: 2026/09/25 15:02:48 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls lb # [ 207.857017] niks3-server[1213]: 2026/09/25 15:02:48 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst lb # [ 207.865022] niks3-server[1213]: 2026/09/25 15:02:48 INFO Received complete push request method=POST path=/api/pushes/82/complete worker1 # [ 207.892098] nix-grpc-daemon[1671]: time=2026-09-25T15:02:48.424Z level=INFO msg="Upload complete. (174ms)" lb # [ 207.869018] niks3-server[1213]: 2026/09/25 15:02:48 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo worker1 # [ 207.898080] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 207.907067] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 207.912007] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 207.915895] nix-daemon[855]: reaped child process 2121, status = succeeded worker1 # [ 207.926064] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 208.784457] nix-daemon[855]: reaped child process 2130, status = succeeded worker1 # [ 208.864358] nix-daemon[855]: accepted connection from pid 2146, user nix-grpc-daemon (trusted) lb # [ 208.860224] niks3-server[1213]: 2026/09/25 15:02:49 INFO Received push request method=POST path=/api/pushes worker1 # [ 208.887190] nix-daemon[855]: reaped child process 2149, status = succeeded worker1 # [ 208.892444] nix-grpc-daemon[1671]: time=2026-09-25T15:02:49.422Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 208.893957] nix-grpc-daemon[1671]: time=2026-09-25T15:02:49.422Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)" lb # [ 208.890664] niks3-server[1213]: 2026/09/25 15:02:49 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv lb # [ 208.899439] niks3-server[1213]: 2026/09/25 15:02:49 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls worker1 # [ 208.926054] nix-grpc-daemon[1671]: time=2026-09-25T15:02:49.457Z level=INFO msg="Uploading 1 narinfos" lb # [ 208.905770] niks3-server[1213]: 2026/09/25 15:02:49 INFO Received sign narinfos request method=POST path=/api/pushes/83/sign lb # [ 208.911916] niks3-server[1213]: 2026/09/25 15:02:49 INFO Signed narinfos id=83 count=1 lb # [ 208.915891] niks3-server[1213]: 2026/09/25 15:02:49 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst lb # [ 208.922668] niks3-server[1213]: 2026/09/25 15:02:49 INFO Received complete push request method=POST path=/api/pushes/83/complete worker1 # [ 208.950750] nix-grpc-daemon[1671]: time=2026-09-25T15:02:49.482Z level=INFO msg="Upload complete. (217ms)" lb # [ 208.928152] niks3-server[1213]: 2026/09/25 15:02:49 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo worker2 # [ 208.929579] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:49Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 208.958354] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=12 outputs=1 worker1 # [ 208.970621] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 208.974716] nix-daemon[855]: reaped child process 2128, status = succeeded worker1 # [ 208.980512] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 208.989172] nix-grpc-daemon[1671]: time=2026-09-25T15:02:49.521Z level=INFO msg="All 2 paths already cached" worker1 # [ 208.995409] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 209.004313] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 209.122357] nix-daemon[855]: accepted connection from pid 2153, user nix-grpc-daemon (trusted) worker1 # [ 209.140419] nix-daemon[855]: reaped child process 2156, status = succeeded lb # [ 209.122831] niks3-server[1213]: 2026/09/25 15:02:49 INFO Received push request method=POST path=/api/pushes worker1 # [ 209.151620] nix-grpc-daemon[1671]: time=2026-09-25T15:02:49.683Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 209.153323] nix-grpc-daemon[1671]: time=2026-09-25T15:02:49.683Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)" lb # [ 209.142527] niks3-server[1213]: 2026/09/25 15:02:49 INFO Received sign narinfos request method=POST path=/api/pushes/84/sign worker1 # [ 209.169377] nix-grpc-daemon[1671]: time=2026-09-25T15:02:49.701Z level=INFO msg="Uploading 1 narinfos" lb # [ 209.149242] niks3-server[1213]: 2026/09/25 15:02:49 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls lb # [ 209.151791] niks3-server[1213]: 2026/09/25 15:02:49 INFO Signed narinfos id=84 count=1 lb # [ 209.161276] niks3-server[1213]: 2026/09/25 15:02:49 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst worker1 # [ 209.189559] nix-grpc-daemon[1671]: time=2026-09-25T15:02:49.721Z level=INFO msg="Upload complete. (177ms)" lb # [ 209.166860] niks3-server[1213]: 2026/09/25 15:02:49 INFO Received complete push request method=POST path=/api/pushes/84/complete lb # [ 209.170579] niks3-server[1213]: 2026/09/25 15:02:49 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo worker1 # [ 209.196181] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 209.200557] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 worker1 # [ 209.208394] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 209.211774] nix-daemon[855]: reaped child process 2151, status = succeeded worker1 # [ 209.221701] nix-daemon[855]: accepted connection from pid 1575, user nix-grpc-daemon (trusted) worker1 # [ 210.501236] nix-daemon[855]: reaped child process 2160, status = succeeded worker1 # [ 210.564260] nix-daemon[855]: accepted connection from pid 2176, user nix-grpc-daemon (trusted) worker1 # [ 210.581539] nix-daemon[855]: reaped child process 2179, status = succeeded lb # [ 210.562697] niks3-server[1213]: 2026/09/25 15:02:51 INFO Received push request method=POST path=/api/pushes worker1 # [ 210.592376] nix-grpc-daemon[1671]: time=2026-09-25T15:02:51.123Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 210.593924] nix-grpc-daemon[1671]: time=2026-09-25T15:02:51.123Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)" lb # [ 210.590579] niks3-server[1213]: 2026/09/25 15:02:51 INFO Received sign narinfos request method=POST path=/api/pushes/85/sign worker1 # [ 210.621167] nix-grpc-daemon[1671]: time=2026-09-25T15:02:51.153Z level=INFO msg="Uploading 1 narinfos" lb # [ 210.599308] niks3-server[1213]: 2026/09/25 15:02:51 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls lb # [ 210.605335] niks3-server[1213]: 2026/09/25 15:02:51 INFO Signed narinfos id=85 count=1 lb # [ 210.609886] niks3-server[1213]: 2026/09/25 15:02:51 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv lb # [ 210.619219] niks3-server[1213]: 2026/09/25 15:02:51 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst worker1 # [ 210.646687] nix-grpc-daemon[1671]: time=2026-09-25T15:02:51.178Z level=INFO msg="Upload complete. (217ms)" lb # [ 210.622637] niks3-server[1213]: 2026/09/25 15:02:51 INFO Received complete push request method=POST path=/api/pushes/85/complete lb # [ 210.627712] niks3-server[1213]: 2026/09/25 15:02:51 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo worker1 # [ 210.659593] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52466 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=13 outputs=1 worker1 # [ 210.664296] nix-daemon[855]: reaped child process 2158, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 12.97 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]' client # [ 210.745192] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]". client # Running as unit: gpu.service; invocation ID: 87edbe81bdf94c4c9ca5211b1aec5c6f client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker2 # [ 210.918514] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:48172 client # [ 210.928695] nix[1576]: don't know how to build these paths: client # [ 210.929631] nix[1576]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv worker1 # [ 210.951904] nix-grpc-daemon[1575]: ts=2026-09-25T15:02:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52466 worker2 # [ 210.938475] nix-grpc-daemon[1910]: ts=2026-09-25T15:02:51Z level=debug event=want client=3 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=0 client # [ 210.944175] nix[1576]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip}, waiting up to 30s for a worker lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) client: waiting for success: systemctl show -p Result gpu | grep -q exit-code lb # [ 218.277256] envoy[897]: conn peer=[2001:db8:1::1]:51206 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27093 tx=12255 ms=12768 client # [ 240.945131] nix[1576]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip} client # [ 240.955068] nix[1576]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed client # [ 240.973713] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE client # [ 240.977687] systemd[1]: gpu.service: Failed with result 'exit-code'. client # [ 240.979602] systemd[1]: gpu.service: Consumed 174ms CPU time over 30.231s wall clock time, 35.4M memory peak, 5.6K incoming IP traffic, 5.2K outgoing IP traffic. client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.24 seconds) client: must succeed: journalctl -u gpu -o cat client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 43.65 seconds) subtest: low disk drains a worker and builds go to the other worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.19 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free' worker1 # [ 243.349733] nix-grpc-daemon[1575]: ts=2026-09-25T15:03:23Z level=info event=unhealthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.07 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2 client # don't know how to build these paths: client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker2 # [ 243.788279] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 243.793319] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 243.808235] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:24Z level=debug event=want client=4 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=0 worker2 # [ 243.811217] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:24Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 243.816874] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 243.827979] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48172 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'... worker2 # [ 243.837669] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 243.846290] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) worker2 # [ 243.972374] nix-daemon[845]: accepted connection from pid 2112, user nix-grpc-daemon (trusted) worker2 # [ 243.994848] nix-daemon[845]: reaped child process 2115, status = succeeded lb # [ 243.997541] niks3-server[1213]: 2026/09/25 15:03:24 INFO Received push request method=POST path=/api/pushes worker2 # [ 244.000879] nix-grpc-daemon[1984]: time=2026-09-25T15:03:24.543Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 244.003130] nix-grpc-daemon[1984]: time=2026-09-25T15:03:24.545Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)" lb # [ 244.016222] niks3-server[1213]: 2026/09/25 15:03:24 INFO Received sign narinfos request method=POST path=/api/pushes/86/sign worker2 # [ 244.018839] nix-grpc-daemon[1984]: time=2026-09-25T15:03:24.561Z level=INFO msg="Uploading 1 narinfos" lb # [ 244.025316] niks3-server[1213]: 2026/09/25 15:03:24 INFO Signed narinfos id=86 count=1 lb # [ 244.029276] niks3-server[1213]: 2026/09/25 15:03:24 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls lb # [ 244.036676] niks3-server[1213]: 2026/09/25 15:03:24 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst lb # [ 244.045229] niks3-server[1213]: 2026/09/25 15:03:24 INFO Received complete push request method=POST path=/api/pushes/86/complete lb # [ 244.051582] niks3-server[1213]: 2026/09/25 15:03:24 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo worker2 # [ 244.052073] nix-grpc-daemon[1984]: time=2026-09-25T15:03:24.594Z level=INFO msg="Upload complete. (196ms)" worker2 # [ 244.058057] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48172 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 244.064826] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 244.076086] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) worker2 # [ 244.078515] nix-daemon[845]: reaped child process 2110, status = succeeded client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051... client # farm-a> worker2-next: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker2 # [ 244.089359] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-drain worker2 # [ 245.152799] nix-daemon[845]: reaped child process 2119, status = succeeded worker2 # [ 245.221057] nix-daemon[845]: accepted connection from pid 2135, user nix-grpc-daemon (trusted) worker2 # [ 245.242868] nix-daemon[845]: reaped child process 2138, status = succeeded lb # [ 245.245008] niks3-server[1213]: 2026/09/25 15:03:25 INFO Received push request method=POST path=/api/pushes worker2 # [ 245.248302] nix-grpc-daemon[1984]: time=2026-09-25T15:03:25.788Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 245.249858] nix-grpc-daemon[1984]: time=2026-09-25T15:03:25.789Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)" lb # [ 245.265002] niks3-server[1213]: 2026/09/25 15:03:25 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls lb # [ 245.273250] niks3-server[1213]: 2026/09/25 15:03:25 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker2 # [ 245.277498] nix-grpc-daemon[1984]: time=2026-09-25T15:03:25.819Z level=INFO msg="Uploading 1 narinfos" lb # [ 245.278513] niks3-server[1213]: 2026/09/25 15:03:25 INFO Received sign narinfos request method=POST path=/api/pushes/87/sign lb # [ 245.284826] niks3-server[1213]: 2026/09/25 15:03:25 INFO Signed narinfos id=87 count=1 lb # [ 245.288872] niks3-server[1213]: 2026/09/25 15:03:25 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst lb # [ 245.296672] niks3-server[1213]: 2026/09/25 15:03:25 INFO Received complete push request method=POST path=/api/pushes/87/complete worker2 # [ 245.301202] nix-grpc-daemon[1984]: time=2026-09-25T15:03:25.843Z level=INFO msg="Upload complete. (203ms)" lb # [ 245.305380] niks3-server[1213]: 2026/09/25 15:03:25 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo worker2 # [ 245.309639] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=14 outputs=1 worker2 # [ 245.325334] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:25Z level=debug event=want client=4 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=0 worker2 # [ 245.328144] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:25Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 245.331105] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 245.334252] nix-daemon[845]: reaped child process 2117, status = succeeded worker2 # [ 245.338914] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 245.346889] nix-grpc-daemon[1984]: time=2026-09-25T15:03:25.889Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'... worker2 # [ 245.353782] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 245.361238] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) worker2 # [ 245.475032] nix-daemon[845]: accepted connection from pid 2144, user nix-grpc-daemon (trusted) lb # [ 245.495591] niks3-server[1213]: 2026/09/25 15:03:26 INFO Received push request method=POST path=/api/pushes worker2 # [ 245.495046] nix-daemon[845]: reaped child process 2147, status = succeeded worker2 # [ 245.497083] nix-grpc-daemon[1984]: time=2026-09-25T15:03:26.039Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 245.498722] nix-grpc-daemon[1984]: time=2026-09-25T15:03:26.039Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)" lb # [ 245.510987] niks3-server[1213]: 2026/09/25 15:03:26 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst worker2 # [ 245.516476] nix-grpc-daemon[1984]: time=2026-09-25T15:03:26.058Z level=INFO msg="Uploading 1 narinfos" lb # [ 245.520563] niks3-server[1213]: 2026/09/25 15:03:26 INFO Received sign narinfos request method=POST path=/api/pushes/88/sign lb # [ 245.523034] niks3-server[1213]: 2026/09/25 15:03:26 INFO Signed narinfos id=88 count=1 lb # [ 245.530369] niks3-server[1213]: 2026/09/25 15:03:26 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls lb # [ 245.537156] niks3-server[1213]: 2026/09/25 15:03:26 INFO Received complete push request method=POST path=/api/pushes/88/complete worker2 # [ 245.540279] nix-grpc-daemon[1984]: time=2026-09-25T15:03:26.082Z level=INFO msg="Upload complete. (170ms)" lb # [ 245.545223] niks3-server[1213]: 2026/09/25 15:03:26 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo worker2 # [ 245.547152] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48172 duration_s=0 paths=1 nar_bytes_in=632 worker2 # [ 245.552211] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 245.563358] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) worker2 # [ 245.565532] nix-daemon[845]: reaped child process 2142, status = succeeded client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051... worker2 # [ 245.574022] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) client # farm-b> worker2-next: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-drain worker2 # [ 246.224850] nix-daemon[845]: reaped child process 2151, status = succeeded worker2 # [ 246.312729] nix-daemon[845]: accepted connection from pid 2167, user nix-grpc-daemon (trusted) worker2 # [ 246.329894] nix-daemon[845]: reaped child process 2170, status = succeeded lb # [ 246.339311] niks3-server[1213]: 2026/09/25 15:03:26 INFO Received push request method=POST path=/api/pushes worker2 # [ 246.343850] nix-grpc-daemon[1984]: time=2026-09-25T15:03:26.883Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 246.345500] nix-grpc-daemon[1984]: time=2026-09-25T15:03:26.883Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)" lb # [ 246.364542] niks3-server[1213]: 2026/09/25 15:03:26 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls worker2 # [ 246.366087] nix-grpc-daemon[1984]: time=2026-09-25T15:03:26.908Z level=INFO msg="Uploading 1 narinfos" lb # [ 246.372460] niks3-server[1213]: 2026/09/25 15:03:26 INFO Received sign narinfos request method=POST path=/api/pushes/89/sign lb # [ 246.378441] niks3-server[1213]: 2026/09/25 15:03:26 INFO Signed narinfos id=89 count=1 lb # [ 246.386331] niks3-server[1213]: 2026/09/25 15:03:26 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker2 # [ 246.386525] nix-grpc-daemon[1984]: time=2026-09-25T15:03:26.929Z level=INFO msg="Upload complete. (203ms)" lb # [ 246.392429] niks3-server[1213]: 2026/09/25 15:03:26 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst lb # [ 246.398187] niks3-server[1213]: 2026/09/25 15:03:26 INFO Received complete push request method=POST path=/api/pushes/89/complete lb # [ 246.401385] niks3-server[1213]: 2026/09/25 15:03:26 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo worker2 # [ 246.395626] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 duration_s=0 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=15 outputs=1 worker2 # [ 246.408335] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:26Z level=debug event=want client=4 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=0 worker2 # [ 246.413137] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:26Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 246.417480] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 246.421002] nix-daemon[845]: reaped child process 2149, status = succeeded worker2 # [ 246.423556] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 246.432434] nix-grpc-daemon[1984]: time=2026-09-25T15:03:26.974Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'... worker2 # [ 246.438887] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 246.446155] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) worker2 # [ 246.557258] nix-daemon[845]: accepted connection from pid 2176, user nix-grpc-daemon (trusted) worker2 # [ 246.575288] nix-daemon[845]: reaped child process 2179, status = succeeded lb # [ 246.584302] niks3-server[1213]: 2026/09/25 15:03:27 INFO Received push request method=POST path=/api/pushes worker2 # [ 246.585786] nix-grpc-daemon[1984]: time=2026-09-25T15:03:27.127Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 246.587338] nix-grpc-daemon[1984]: time=2026-09-25T15:03:27.128Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)" lb # [ 246.601661] niks3-server[1213]: 2026/09/25 15:03:27 INFO Received sign narinfos request method=POST path=/api/pushes/90/sign worker2 # [ 246.601391] nix-grpc-daemon[1984]: time=2026-09-25T15:03:27.143Z level=INFO msg="Uploading 1 narinfos" lb # [ 246.609279] niks3-server[1213]: 2026/09/25 15:03:27 INFO Signed narinfos id=90 count=1 lb # [ 246.611005] niks3-server[1213]: 2026/09/25 15:03:27 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst lb # [ 246.621943] niks3-server[1213]: 2026/09/25 15:03:27 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls worker2 # [ 246.620849] nix-grpc-daemon[1984]: time=2026-09-25T15:03:27.163Z level=INFO msg="Upload complete. (165ms)" lb # [ 246.628057] niks3-server[1213]: 2026/09/25 15:03:27 INFO Received complete push request method=POST path=/api/pushes/90/complete lb # [ 246.630485] niks3-server[1213]: 2026/09/25 15:03:27 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo worker2 # [ 246.628108] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48172 duration_s=0 paths=1 nar_bytes_in=760 worker2 # [ 246.632080] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 worker2 # [ 246.639640] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) worker2 # [ 246.643220] nix-daemon[845]: reaped child process 2174, status = succeeded client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051... worker2 # [ 246.651731] nix-daemon[845]: accepted connection from pid 1910, user nix-grpc-daemon (trusted) client # farm-top> worker2-next: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-drain worker2 # [ 247.891029] nix-daemon[845]: reaped child process 2183, status = succeeded worker2 # [ 247.987515] nix-daemon[845]: accepted connection from pid 2199, user nix-grpc-daemon (trusted) worker2 # [ 248.002207] nix-daemon[845]: reaped child process 2202, status = succeeded lb # [ 248.012960] niks3-server[1213]: 2026/09/25 15:03:28 INFO Received push request method=POST path=/api/pushes worker2 # [ 248.016399] nix-grpc-daemon[1984]: time=2026-09-25T15:03:28.556Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 248.017910] nix-grpc-daemon[1984]: time=2026-09-25T15:03:28.556Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)" lb # [ 248.038902] niks3-server[1213]: 2026/09/25 15:03:28 INFO Received sign narinfos request method=POST path=/api/pushes/91/sign worker2 # [ 248.040298] nix-grpc-daemon[1984]: time=2026-09-25T15:03:28.582Z level=INFO msg="Uploading 1 narinfos" lb # [ 248.046572] niks3-server[1213]: 2026/09/25 15:03:28 INFO Signed narinfos id=91 count=1 lb # [ 248.051518] niks3-server[1213]: 2026/09/25 15:03:28 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv lb # [ 248.058940] niks3-server[1213]: 2026/09/25 15:03:28 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls lb # [ 248.065949] niks3-server[1213]: 2026/09/25 15:03:28 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst worker2 # [ 248.068607] nix-grpc-daemon[1984]: time=2026-09-25T15:03:28.611Z level=INFO msg="Upload complete. (207ms)" lb # [ 248.073576] niks3-server[1213]: 2026/09/25 15:03:28 INFO Received complete push request method=POST path=/api/pushes/91/complete lb # [ 248.077592] niks3-server[1213]: 2026/09/25 15:03:28 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo worker2 # [ 248.077250] nix-grpc-daemon[1910]: ts=2026-09-25T15:03:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48172 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=16 outputs=1 worker2 # [ 248.087125] nix-daemon[845]: reaped child process 2181, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 4.49 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.14 seconds) lb # [ 248.282384] envoy[897]: conn peer=[2001:db8:1::1]:55598 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=931 tx=612 ms=30038 lb # [ 248.285155] envoy[897]: conn peer=[2001:db8:1::1]:33056 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9485 tx=4349 ms=4303 worker1: must succeed: rm /nix/.rw-store/fill worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.12 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free' worker1 # [ 249.358227] nix-grpc-daemon[1575]: ts=2026-09-25T15:03:29Z level=info event=healthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.07 seconds) (finished: subtest: low disk drains a worker and builds go to the other, in 8.13 seconds) (finished: run the VM test script, in 250.45 seconds) test script finished in 250.59s cleanup kill QemuMachine (pid 49) client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) client # [2026-09-25T15:03:30Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-25T15:03:30Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-25T15:03:30Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 50) lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) lb # [2026-09-25T15:03:30Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-25T15:03:30Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-25T15:03:30Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 51) worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker1 # [2026-09-25T15:03:30Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-25T15:03:30Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-25T15:03:30Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 52) worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker2 # [2026-09-25T15:03:31Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-25T15:03:31Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-25T15:03:31Z INFO virtiofsd] Client disconnected, shutting down (finished: cleanup, in 1.04 seconds) additionally exposed symbols: client, lb, worker1, worker2, vlan1, start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh