tribuchet: building on eliza Machine state will be reset. To keep it, pass --keep-machine-state start all VLans (finished: start all VLans, in 0.00 seconds) Test will time out and terminate in 900.0 seconds run the VM test script start all VMs client: starting vm lb: starting vm worker1: starting vm worker2: starting vm client: QEMU running (pid 50) client # Disk image does not exist, creating the virtualisation disk image... worker1 # Disk image does not exist, creating the virtualisation disk image... client # Formatting '/build/vm-state-client/tmp.JO3hjrkQ10', fmt=raw size=1073741824 worker2 # Disk image does not exist, creating the virtualisation disk image... lb # Disk image does not exist, creating the virtualisation disk image... lb: QEMU running (pid 49) client # mke2fs 1.47.4 (6-Mar-2025) worker2 # Formatting '/build/vm-state-worker2/tmp.iH2VnOHPtG', fmt=raw size=1073741824 lb # Formatting '/build/vm-state-lb/tmp.KRPi5SeysP', fmt=raw size=1073741824 worker1 # Formatting '/build/vm-state-worker1/tmp.k2DZKaqZEx', fmt=raw size=1073741824 worker1: QEMU running (pid 51) worker2: QEMU running (pid 52) client # Discarding device blocks: 0/262144 done worker2 # mke2fs 1.47.4 (6-Mar-2025) worker1 # mke2fs 1.47.4 (6-Mar-2025) lb # mke2fs 1.47.4 (6-Mar-2025) client # Creating filesystem with 262144 4k blocks and 65536 inodes worker1 # Discarding device blocks: 0/262144 done lb # Discarding device blocks: 0/262144 done client # Filesystem UUID: 9b461f11-b1d6-4956-868d-e04030333ae3 worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes lb # Creating filesystem with 262144 4k blocks and 65536 inodes client # Superblock backups stored on blocks: worker1 # Filesystem UUID: c161b43a-11e9-4d85-b10c-52abfa81c9ee lb # Filesystem UUID: 8622face-eab8-4b8c-b722-453650c9cee3 client # 32768, 98304, 163840, 229376 worker1 # Superblock backups stored on blocks: lb # Superblock backups stored on blocks: client # worker1 # 32768, 98304, 163840, 229376 lb # 32768, 98304, 163840, 229376 client # Allocating group tables: 0/8 done worker1 # lb # client # Writing inode tables: 0/8 done worker1 # Allocating group tables: 0/8 done lb # Allocating group tables: 0/8 done client # Creating journal (8192 blocks): done worker1 # Writing inode tables: 0/8 done lb # Writing inode tables: 0/8 done client # Writing superblocks and filesystem accounting information: 0/8 done worker1 # Creating journal (8192 blocks): done lb # Creating journal (8192 blocks): done client # worker1 # Writing superblocks and filesystem accounting information: 0/8 done lb # Writing superblocks and filesystem accounting information: 0/8 done client # Virtualisation disk image created. worker1 # lb # client # Starting virtiofs daemons... worker1 # Virtualisation disk image created. lb # Virtualisation disk image created. client # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # Starting virtiofs daemons... lb # Starting virtiofs daemons... client # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-23T04:42:20Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-23T04:42:20Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-23T04:42:20Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-23T04:42:20Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-23T04:42:20Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-23T04:42:20Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T04:42:20Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-23T04:42:20Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T04:42:20Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-23T04:42:20Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-23T04:42:20Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T04:42:20Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-23T04:42:20Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-23T04:42:20Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T04:42:20Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-23T04:42:20Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-23T04:42:20Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T04:42:20Z INFO virtiofsd] Client connected, servicing requests (finished: start all VMs, in 0.57 seconds) worker2 # Discarding device blocks: 0/262144 done client: waiting for unit network-addresses-eth1.service worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes client: waiting for the VM to finish booting worker2 # Filesystem UUID: 37680bca-f3e2-43ad-91ac-3b964e5ce3f5 worker2 # Superblock backups stored on blocks: worker2 # 32768, 98304, 163840, 229376 worker2 # worker2 # Allocating group tables: 0/8 done worker2 # Writing inode tables: 0/8 done worker2 # Creating journal (8192 blocks): done worker2 # Writing superblocks and filesystem accounting information: 0/8 done worker2 # worker2 # Virtualisation disk image created. worker2 # Starting virtiofs daemons... worker2 # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-23T04:42:20Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-23T04:42:20Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-23T04:42:20Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-23T04:42:20Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-23T04:42:20Z INFO virtiofsd] Client connected, servicing requests worker2 # [2026-09-23T04:42:20Z INFO virtiofsd] Client connected, servicing requests worker2 # [2026-09-23T04:42:20Z INFO virtiofsd] Client connected, servicing requests worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker2 # [ 0.000000] KASLR enabled worker2 # [ 0.000000] random: crng init done worker2 # [ 0.000000] Machine model: linux,dummy-virt worker2 # [ 0.000000] efi: UEFI not found. worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker2 # [ 0.000000] Zone ranges: worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] DMA32 empty worker2 # [ 0.000000] Normal empty worker2 # [ 0.000000] Device empty worker2 # [ 0.000000] Movable zone start for each node worker2 # [ 0.000000] Early memory node ranges worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker2 # [ 0.000000] psci: probing for conduit method from DT. worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker2 # [ 0.000000] psci: Trusted OS migration not required worker2 # [ 0.000000] psci: SMC Calling Convention v1.1 worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker1 # [ 0.000000] KASLR enabled worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.000000] random: crng init done worker1 # [ 0.000000] Machine model: linux,dummy-virt worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker1 # [ 0.000000] efi: UEFI not found. worker2 # [ 0.000000] Detected PIPT I-cache on CPU0 worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker2 # [ 0.000000] CPU features: detected: Spectre-v4 worker1 # [ 0.000000] Zone ranges: client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker2 # [ 0.000000] CPU features: detected: Spectre-BHB worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] DMA32 empty worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker1 # [ 0.000000] Normal empty worker1 # [ 0.000000] Device empty client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 client # [ 0.000000] KASLR enabled worker1 # [ 0.000000] Movable zone start for each node worker2 # [ 0.000000] alternatives: applying boot alternatives client # [ 0.000000] random: crng init done worker1 # [ 0.000000] Early memory node ranges client # [ 0.000000] Machine model: linux,dummy-virt client # [ 0.000000] efi: UEFI not found. worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] psci: probing for conduit method from DT. client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware. client # [ 0.000000] Zone ranges: worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/rcm8j4pnk0zmlmgsymh11jg6bfkli2cx-nixos-system-worker2-test/init regInfo=/nix/store/fvbwcpswlxlq4vwgzpa0axny9f1av68s-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] psci: Trusted OS migration not required client # [ 0.000000] DMA32 empty client # [ 0.000000] Normal empty worker1 # [ 0.000000] psci: SMC Calling Convention v1.1 worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/fvbwcpswlxlq4vwgzpa0axny9f1av68s-closure-info/registration", will be passed to user space. client # [ 0.000000] Device empty client # [ 0.000000] Movable zone start for each node worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) client # [ 0.000000] Early memory node ranges worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] Detected PIPT I-cache on CPU0 client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface client # [ 0.000000] psci: probing for conduit method from DT. worker2 # [ 0.000000] software IO TLB: area num 1. worker1 # [ 0.000000] CPU features: detected: Spectre-v4 client # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker1 # [ 0.000000] CPU features: detected: Spectre-BHB client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) client # [ 0.000000] psci: Trusted OS migration not required worker2 # [ 0.000000] Fallback order for Node 0: 0 client # [ 0.000000] psci: SMC Calling Convention v1.1 worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker1 # [ 0.000000] alternatives: applying boot alternatives worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker2 # [ 0.000000] Policy zone: DMA client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 client # [ 0.000000] Detected PIPT I-cache on CPU0 worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker2 # [ 0.000000] allocated 3145728 bytes of page_ext client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages client # [ 0.000000] CPU features: detected: GICv3 CPU interface worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/bn37sc2n1i20ygvgim4ix1wq2wbb4gsw-nixos-system-worker1-test/init regInfo=/nix/store/a7bckkg7lmpj6xb6ww918qxp6p1axf1r-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups client # [ 0.000000] CPU features: detected: Spectre-v4 worker2 # [ 0.000000] rcu: Hierarchical RCU implementation. client # [ 0.000000] CPU features: detected: Spectre-BHB worker2 # [ 0.000000] rcu: RCU event tracing is enabled. client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/a7bckkg7lmpj6xb6ww918qxp6p1axf1r-closure-info/registration", will be passed to user space. worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled. client # [ 0.000000] alternatives: applying boot alternatives worker2 # [ 0.000000] Rude variant of Tasks RCU enabled. worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 worker1 # [ 0.000000] software IO TLB: area num 1. worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/h0h42468m1lycnnbixi8y4srnpd3qprr-nixos-system-client-test/init regInfo=/nix/store/kglfpnzighnmmpghycx27bi8sqvfg32g-closure-info/registration console=ttyAMA0,115200n8 console=tty0 lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 lb # [ 0.000000] KASLR enabled lb # [ 0.000000] random: crng init done worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] Machine model: linux,dummy-virt worker1 # [ 0.000000] Fallback order for Node 0: 0 lb # [ 0.000000] efi: UEFI not found. client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/kglfpnzighnmmpghycx27bi8sqvfg32g-closure-info/registration", will be passed to user space. worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker1 # [ 0.000000] Policy zone: DMA client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker2 # [ 0.000000] GICv3: 256 SPIs implemented worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq lb # [ 0.000000] Zone ranges: client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker1 # [ 0.000000] allocated 3145728 bytes of page_ext worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 lb # [ 0.000000] DMA32 empty client # [ 0.000000] software IO TLB: area num 1. lb # [ 0.000000] Normal empty worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups lb # [ 0.000000] Device empty worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker1 # [ 0.000000] rcu: Hierarchical RCU implementation. client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB lb # [ 0.000000] Movable zone start for each node worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker1 # [ 0.000000] rcu: RCU event tracing is enabled. lb # [ 0.000000] Early memory node ranges client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1) lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] Fallback order for Node 0: 0 worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled. lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker1 # [ 0.000000] Rude variant of Tasks RCU enabled. worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1) client # [ 0.000000] Policy zone: DMA lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled. worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a00000 client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off lb # [ 0.000000] psci: probing for conduit method from DT. worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. lb # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a10000 client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 client # [ 0.000000] allocated 3145728 bytes of page_ext worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. lb # [ 0.000000] psci: Trusted OS migration not required client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] psci: SMC Calling Convention v1.1 client # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns client # [ 0.000000] rcu: Hierarchical RCU implementation. lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). client # [ 0.000000] rcu: RCU event tracing is enabled. lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. lb # [ 0.000000] Detected PIPT I-cache on CPU0 worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 client # [ 0.000000] Trampoline variant of Tasks RCU enabled. worker1 # [ 0.000000] GICv3: 256 SPIs implemented lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) client # [ 0.000000] Rude variant of Tasks RCU enabled. worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented worker2 # [ 0.000034] arm-pv: using stolen time PV lb # [ 0.000000] CPU features: detected: GICv3 CPU interface client # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq lb # [ 0.000000] CPU features: detected: Spectre-v4 worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. lb # [ 0.000000] CPU features: detected: Spectre-BHB worker2 # [ 0.000669] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker2 # [ 0.000860] Console: colour dummy device 80x25 client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker2 # [ 0.000868] printk: legacy console [tty0] enabled lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] alternatives: applying boot alternatives worker2 # [ 0.001070] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.001077] pid_max: default: 32768 minimum: 301 worker2 # [ 0.001152] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.001314] landlock: Up and running. client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker2 # [ 0.001317] Yama: becoming mindful. worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 client # [ 0.000000] GICv3: 256 SPIs implemented worker2 # [ 0.001985] LSM support for eBPF active worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 client # [ 0.000000] GICv3: 0 Extended SPIs implemented lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/kgjzg78npi7z07am2szqxkr41f06jp5w-nixos-system-lb-test/init regInfo=/nix/store/lcw0xq38k7svc2ifmp8wza1s56mbyq61-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker2 # [ 0.002171] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.000000] Root IRQ handler: gic_handle_irq worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI worker2 # [ 0.002207] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker2 # [ 0.004254] rcu: Hierarchical SRCU implementation. lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/lcw0xq38k7svc2ifmp8wza1s56mbyq61-closure-info/registration", will be passed to user space. worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker2 # [ 0.004259] rcu: Max phase no-delay instances is 1000. client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker2 # [ 0.005545] fsl-mc MSI: its@8080000 domain created lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker2 # [ 0.005638] EFI services will not be available. worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker2 # [ 0.005721] smp: Bringing up secondary CPUs ... lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker2 # [ 0.005730] smp: Brought up 1 node, 1 CPU worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker2 # [ 0.005733] SMP: Total of 1 processors activated. worker1 # [ 0.000033] arm-pv: using stolen time PV worker2 # [ 0.005739] CPU: All CPU(s) started at EL1 lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker2 # [ 0.005750] CPU features: detected: Branch Target Identification worker1 # [ 0.000454] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) lb # [ 0.000000] software IO TLB: area num 1. client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker1 # [ 0.000670] Console: colour dummy device 80x25 worker2 # [ 0.005755] CPU features: detected: ARMv8.4 Translation Table Level lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker1 # [ 0.000678] printk: legacy console [tty0] enabled client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker2 # [ 0.005758] CPU features: detected: Instruction cache invalidation not required for I/D coherence lb # [ 0.000000] Fallback order for Node 0: 0 worker1 # [ 0.000891] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker2 # [ 0.005762] CPU features: detected: Data cache clean to the PoU not required for I/D coherence lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker1 # [ 0.000898] pid_max: default: 32768 minimum: 301 client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). lb # [ 0.000000] Policy zone: DMA worker2 # [ 0.005765] CPU features: detected: Common not Private translations worker1 # [ 0.000977] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker1 # [ 0.001125] landlock: Up and running. lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker2 # [ 0.005768] CPU features: detected: CRC32 instructions client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker1 # [ 0.001128] Yama: becoming mindful. lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker1 # [ 0.001610] LSM support for eBPF active worker2 # [ 0.005771] CPU features: detected: Data cache clean to Point of Deep Persistence client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns lb # [ 0.000000] allocated 3145728 bytes of page_ext worker1 # [ 0.001754] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.000032] arm-pv: using stolen time PV worker2 # [ 0.005775] CPU features: detected: Data cache clean to Point of Persistence lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker1 # [ 0.001789] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.005778] CPU features: detected: Data independent timing control (DIT) client # [ 0.000471] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker2 # [ 0.005781] CPU features: detected: E0PD lb # [ 0.000000] rcu: Hierarchical RCU implementation. worker1 # [ 0.003852] rcu: Hierarchical SRCU implementation. client # [ 0.000676] Console: colour dummy device 80x25 lb # [ 0.000000] rcu: RCU event tracing is enabled. worker1 # [ 0.003858] rcu: Max phase no-delay instances is 1000. worker2 # [ 0.005784] CPU features: detected: Enhanced Counter Virtualization client # [ 0.000684] printk: legacy console [tty0] enabled worker1 # [ 0.005181] fsl-mc MSI: its@8080000 domain created lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker2 # [ 0.005787] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) worker1 # [ 0.005271] EFI services will not be available. lb # [ 0.000000] Trampoline variant of Tasks RCU enabled. client # [ 0.000894] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker1 # [ 0.005346] smp: Bringing up secondary CPUs ... worker2 # [ 0.005790] CPU features: detected: Enhanced Virtualization Traps lb # [ 0.000000] Rude variant of Tasks RCU enabled. worker1 # [ 0.005355] smp: Brought up 1 node, 1 CPU client # [ 0.000902] pid_max: default: 32768 minimum: 301 worker2 # [ 0.005794] CPU features: detected: Fine Grained Traps lb # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.005358] SMP: Total of 1 processors activated. client # [ 0.000988] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker1 # [ 0.005364] CPU: All CPU(s) started at EL1 worker2 # [ 0.005797] CPU features: detected: Generic authentication (architected QARMA5 algorithm) client # [ 0.001140] landlock: Up and running. lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. client # [ 0.001144] Yama: becoming mindful. worker2 # [ 0.005802] CPU features: detected: RCpc load-acquire (LDAPR) worker1 # [ 0.005375] CPU features: detected: Branch Target Identification lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 client # [ 0.001630] LSM support for eBPF active worker2 # [ 0.005805] CPU features: detected: LSE atomic instructions worker1 # [ 0.005380] CPU features: detected: ARMv8.4 Translation Table Level client # [ 0.001799] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.005808] CPU features: detected: Privileged Access Never worker1 # [ 0.005383] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker2 # [ 0.005810] CPU features: detected: PMUv3 client # [ 0.001834] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.005813] CPU features: detected: RAS Extension Support client # [ 0.003764] rcu: Hierarchical SRCU implementation. worker1 # [ 0.005387] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker2 # [ 0.005816] CPU features: detected: RASv1p1 Extension Support lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.003770] rcu: Max phase no-delay instances is 1000. worker1 # [ 0.005391] CPU features: detected: Common not Private translations worker2 # [ 0.005819] CPU features: detected: Random Number Generator client # [ 0.005052] fsl-mc MSI: its@8080000 domain created lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker1 # [ 0.005394] CPU features: detected: CRC32 instructions client # [ 0.005145] EFI services will not be available. lb # [ 0.000000] GICv3: 256 SPIs implemented worker2 # [ 0.005821] CPU features: detected: Speculation barrier (SB) client # [ 0.005216] smp: Bringing up secondary CPUs ... lb # [ 0.000000] GICv3: 0 Extended SPIs implemented worker1 # [ 0.005397] CPU features: detected: Data cache clean to Point of Deep Persistence worker2 # [ 0.005824] CPU features: detected: Stage-2 Force Write-Back client # [ 0.005225] smp: Brought up 1 node, 1 CPU lb # [ 0.000000] Root IRQ handler: gic_handle_irq worker1 # [ 0.005400] CPU features: detected: Data cache clean to Point of Persistence worker2 # [ 0.005827] CPU features: detected: TLB range maintenance instructions client # [ 0.005228] SMP: Total of 1 processors activated. lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI client # [ 0.005233] CPU: All CPU(s) started at EL1 lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker1 # [ 0.005403] CPU features: detected: Data independent timing control (DIT) worker2 # [ 0.005832] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker1 # [ 0.005407] CPU features: detected: E0PD client # [ 0.005244] CPU features: detected: Branch Target Identification worker2 # [ 0.005870] alternatives: applying system-wide alternatives lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker1 # [ 0.005409] CPU features: detected: Enhanced Counter Virtualization client # [ 0.005249] CPU features: detected: ARMv8.4 Translation Table Level lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker2 # [ 0.008920] CPU features: detected: BBM Level 2 without TLB conflict abort worker1 # [ 0.005412] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) client # [ 0.005252] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker1 # [ 0.005416] CPU features: detected: Enhanced Virtualization Traps worker2 # [ 0.009113] Memory: 1405916K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved) worker1 # [ 0.005419] CPU features: detected: Fine Grained Traps client # [ 0.005255] CPU features: detected: Data cache clean to the PoU not required for I/D coherence lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker2 # [ 0.009535] devtmpfs: initialized client # [ 0.005259] CPU features: detected: Common not Private translations worker1 # [ 0.005422] CPU features: detected: Generic authentication (architected QARMA5 algorithm) lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker2 # [ 0.011515] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) client # [ 0.005262] CPU features: detected: CRC32 instructions worker1 # [ 0.005427] CPU features: detected: RCpc load-acquire (LDAPR) lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker2 # [ 0.011538] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker1 # [ 0.005430] CPU features: detected: LSE atomic instructions client # [ 0.005265] CPU features: detected: Data cache clean to Point of Deep Persistence lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. worker1 # [ 0.005432] CPU features: detected: Privileged Access Never worker2 # [ 0.011734] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL client # [ 0.005269] CPU features: detected: Data cache clean to Point of Persistence worker1 # [ 0.005435] CPU features: detected: PMUv3 worker2 # [ 0.011738] 0 pages in range for non-PLT usage lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns client # [ 0.005272] CPU features: detected: Data independent timing control (DIT) worker2 # [ 0.011739] 508272 pages in range for PLT usage worker1 # [ 0.005438] CPU features: detected: RAS Extension Support lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). client # [ 0.005275] CPU features: detected: E0PD worker2 # [ 0.011858] pinctrl core: initialized pinctrl subsystem worker1 # [ 0.005441] CPU features: detected: RASv1p1 Extension Support worker2 # [ 0.012628] DMI not present or invalid. client # [ 0.005278] CPU features: detected: Enhanced Counter Virtualization worker1 # [ 0.005444] CPU features: detected: Random Number Generator lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker2 # [ 0.016400] NET: Registered PF_NETLINK/PF_ROUTE protocol family client # [ 0.005281] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) worker1 # [ 0.005446] CPU features: detected: Speculation barrier (SB) worker2 # [ 0.018796] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns client # [ 0.005284] CPU features: detected: Enhanced Virtualization Traps worker1 # [ 0.005449] CPU features: detected: Stage-2 Force Write-Back lb # [ 0.000031] arm-pv: using stolen time PV client # [ 0.005288] CPU features: detected: Fine Grained Traps worker1 # [ 0.005452] CPU features: detected: TLB range maintenance instructions worker2 # [ 0.018981] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations lb # [ 0.000377] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker1 # [ 0.005457] CPU features: detected: Speculative Store Bypassing Safe (SSBS) client # [ 0.005291] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker2 # [ 0.019186] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations lb # [ 0.000567] Console: colour dummy device 80x25 worker1 # [ 0.005496] alternatives: applying system-wide alternatives client # [ 0.005296] CPU features: detected: RCpc load-acquire (LDAPR) worker2 # [ 0.019211] audit: initializing netlink subsys (disabled) lb # [ 0.000574] printk: legacy console [tty0] enabled client # [ 0.005299] CPU features: detected: LSE atomic instructions worker1 # [ 0.008580] CPU features: detected: BBM Level 2 without TLB conflict abort worker2 # [ 0.019836] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.005302] CPU features: detected: Privileged Access Never worker2 # [ 0.019838] thermal_sys: Registered thermal governor 'bang_bang' lb # [ 0.000764] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) client # [ 0.005305] CPU features: detected: PMUv3 lb # [ 0.000770] pid_max: default: 32768 minimum: 301 worker2 # [ 0.019842] thermal_sys: Registered thermal governor 'step_wise' worker1 # [ 0.008741] Memory: 1405636K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved) client # [ 0.005307] CPU features: detected: RAS Extension Support worker1 # [ 0.009205] devtmpfs: initialized lb # [ 0.000874] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker2 # [ 0.019845] thermal_sys: Registered thermal governor 'user_space' client # [ 0.005310] CPU features: detected: RASv1p1 Extension Support lb # [ 0.001018] landlock: Up and running. worker1 # [ 0.011028] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) lb # [ 0.001021] Yama: becoming mindful. worker2 # [ 0.019850] thermal_sys: Registered thermal governor 'power_allocator' client # [ 0.005313] CPU features: detected: Random Number Generator lb # [ 0.001455] LSM support for eBPF active client # [ 0.005316] CPU features: detected: Speculation barrier (SB) worker1 # [ 0.011052] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker2 # [ 0.019877] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 lb # [ 0.001609] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.005318] CPU features: detected: Stage-2 Force Write-Back worker2 # [ 0.019884] cpuidle: using governor ladder worker1 # [ 0.011265] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL worker2 # [ 0.019890] cpuidle: using governor menu worker1 # [ 0.011269] 0 pages in range for non-PLT usage client # [ 0.005321] CPU features: detected: TLB range maintenance instructions lb # [ 0.001647] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker1 # [ 0.011270] 508272 pages in range for PLT usage worker2 # [ 0.020089] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. lb # [ 0.003440] rcu: Hierarchical SRCU implementation. client # [ 0.005326] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker1 # [ 0.011399] pinctrl core: initialized pinctrl subsystem worker2 # [ 0.020105] ASID allocator initialised with 65536 entries lb # [ 0.003445] rcu: Max phase no-delay instances is 1000. client # [ 0.005365] alternatives: applying system-wide alternatives worker1 # [ 0.012147] DMI not present or invalid. worker2 # [ 0.021404] Serial: AMBA PL011 UART driver lb # [ 0.004688] fsl-mc MSI: its@8080000 domain created client # [ 0.008456] CPU features: detected: BBM Level 2 without TLB conflict abort worker1 # [ 0.015731] NET: Registered PF_NETLINK/PF_ROUTE protocol family lb # [ 0.004788] EFI services will not be available. worker2 # [ 0.026673] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 lb # [ 0.004854] smp: Bringing up secondary CPUs ... worker1 # [ 0.018039] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker2 # [ 0.026812] printk: console [ttyAMA0] enabled lb # [ 0.004862] smp: Brought up 1 node, 1 CPU lb # [ 0.004865] SMP: Total of 1 processors activated. client # [ 0.008639] Memory: 1405856K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved) worker1 # [ 0.018235] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations client # [ 0.009045] devtmpfs: initialized lb # [ 0.004870] CPU: All CPU(s) started at EL1 worker1 # [ 0.018444] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations lb # [ 0.004881] CPU features: detected: Branch Target Identification client # [ 0.010957] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker1 # [ 0.018471] audit: initializing netlink subsys (disabled) lb # [ 0.004886] CPU features: detected: ARMv8.4 Translation Table Level client # [ 0.010981] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker1 # [ 0.019084] thermal_sys: Registered thermal governor 'fair_share' lb # [ 0.004889] CPU features: detected: Instruction cache invalidation not required for I/D coherence client # [ 0.011183] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL worker1 # [ 0.019087] thermal_sys: Registered thermal governor 'bang_bang' client # [ 0.011187] 0 pages in range for non-PLT usage worker1 # [ 0.019090] thermal_sys: Registered thermal governor 'step_wise' lb # [ 0.004893] CPU features: detected: Data cache clean to the PoU not required for I/D coherence client # [ 0.011189] 508272 pages in range for PLT usage worker1 # [ 0.019093] thermal_sys: Registered thermal governor 'user_space' client # [ 0.011317] pinctrl core: initialized pinctrl subsystem lb # [ 0.004896] CPU features: detected: Common not Private translations client # [ 0.012121] DMI not present or invalid. worker1 # [ 0.019099] thermal_sys: Registered thermal governor 'power_allocator' lb # [ 0.004900] CPU features: detected: CRC32 instructions client # [ 0.015749] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker1 # [ 0.019134] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 lb # [ 0.004902] CPU features: detected: Data cache clean to Point of Deep Persistence worker1 # [ 0.019142] cpuidle: using governor ladder client # [ 0.018053] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker2 # [ 0.145267] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages lb # [ 0.004906] CPU features: detected: Data cache clean to Point of Persistence worker1 # [ 0.019149] cpuidle: using governor menu worker2 # [ 0.145286] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page client # [ 0.018249] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations lb # [ 0.004909] CPU features: detected: Data independent timing control (DIT) worker1 # [ 0.019350] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker2 # [ 0.145291] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages lb # [ 0.004912] CPU features: detected: E0PD worker1 # [ 0.019366] ASID allocator initialised with 65536 entries client # [ 0.018455] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker1 # [ 0.020573] Serial: AMBA PL011 UART driver lb # [ 0.004914] CPU features: detected: Enhanced Counter Virtualization worker2 # [ 0.145295] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page client # [ 0.018481] audit: initializing netlink subsys (disabled) lb # [ 0.004918] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) client # [ 0.019090] thermal_sys: Registered thermal governor 'fair_share' worker2 # [ 0.145299] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker1 # [ 0.025864] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 worker1 # [ 0.026036] printk: console [ttyAMA0] enabled lb # [ 0.004922] CPU features: detected: Enhanced Virtualization Traps worker2 # [ 0.145303] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page client # [ 0.019093] thermal_sys: Registered thermal governor 'bang_bang' lb # [ 0.004925] CPU features: detected: Fine Grained Traps client # [ 0.019096] thermal_sys: Registered thermal governor 'step_wise' worker2 # [ 0.145308] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages client # [ 0.019099] thermal_sys: Registered thermal governor 'user_space' lb # [ 0.004928] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker2 # [ 0.145312] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page lb # [ 0.004932] CPU features: detected: RCpc load-acquire (LDAPR) client # [ 0.019107] thermal_sys: Registered thermal governor 'power_allocator' lb # [ 0.004935] CPU features: detected: LSE atomic instructions client # [ 0.019141] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 lb # [ 0.004938] CPU features: detected: Privileged Access Never client # [ 0.019150] cpuidle: using governor ladder worker2 # [ 0.152693] fbcon: Taking over console lb # [ 0.004941] CPU features: detected: PMUv3 client # [ 0.019156] cpuidle: using governor menu worker2 # [ 0.152709] ACPI: Interpreter disabled. lb # [ 0.004944] CPU features: detected: RAS Extension Support client # [ 0.019339] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. lb # [ 0.004946] CPU features: detected: RASv1p1 Extension Support client # [ 0.019356] ASID allocator initialised with 65536 entries lb # [ 0.004949] CPU features: detected: Random Number Generator client # [ 0.020633] Serial: AMBA PL011 UART driver lb # [ 0.004951] CPU features: detected: Speculation barrier (SB) client # [ 0.026026] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 lb # [ 0.004954] CPU features: detected: Stage-2 Force Write-Back client # [ 0.026158] printk: console [ttyAMA0] enabled worker1 # [ 0.143310] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages lb # [ 0.004957] CPU features: detected: TLB range maintenance instructions worker1 # [ 0.143330] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page lb # [ 0.004962] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker1 # [ 0.143335] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages lb # [ 0.004999] alternatives: applying system-wide alternatives worker1 # [ 0.143340] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page lb # [ 0.007921] CPU features: detected: BBM Level 2 without TLB conflict abort worker1 # [ 0.143344] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker1 # [ 0.143349] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.008212] Memory: 1405672K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved) worker1 # [ 0.143353] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages worker2 # [ 0.161727] iommu: Default domain type: Translated lb # [ 0.008599] devtmpfs: initialized worker1 # [ 0.143358] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker2 # [ 0.161739] iommu: DMA domain TLB invalidation policy: strict mode lb # [ 0.010401] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker2 # [ 0.162102] SCSI subsystem initialized lb # [ 0.010423] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). client # [ 0.144529] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker2 # [ 0.164106] usbcore: registered new interface driver usbfs lb # [ 0.010602] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL worker1 # [ 0.150858] fbcon: Taking over console worker2 # [ 0.164144] usbcore: registered new interface driver hub client # [ 0.144549] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker1 # [ 0.150874] ACPI: Interpreter disabled. lb # [ 0.010606] 0 pages in range for non-PLT usage worker2 # [ 0.164160] usbcore: registered new device driver usb lb # [ 0.010607] 508272 pages in range for PLT usage client # [ 0.144555] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages worker2 # [ 0.164424] pps_core: LinuxPPS API ver. 1 registered lb # [ 0.010701] pinctrl core: initialized pinctrl subsystem client # [ 0.144559] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page lb # [ 0.011485] DMI not present or invalid. worker2 # [ 0.164431] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti client # [ 0.144564] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages lb # [ 0.014933] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker2 # [ 0.164441] PTP clock support registered worker2 # [ 0.164489] EDAC MC: Ver: 3.0.0 client # [ 0.144568] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.017229] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations client # [ 0.144572] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages worker2 # [ 0.169082] scmi_core: SCMI protocol bus registered lb # [ 0.017402] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations client # [ 0.144577] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker2 # [ 0.170064] FPGA manager framework lb # [ 0.017602] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker2 # [ 0.171063] vgaarb: loaded lb # [ 0.017627] audit: initializing netlink subsys (disabled) worker2 # [ 0.171719] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.018220] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.152093] fbcon: Taking over console client # [ 0.152110] ACPI: Interpreter disabled. lb # [ 0.018222] thermal_sys: Registered thermal governor 'bang_bang' lb # [ 0.018225] thermal_sys: Registered thermal governor 'step_wise' worker1 # [ 0.159887] iommu: Default domain type: Translated lb # [ 0.018228] thermal_sys: Registered thermal governor 'user_space' worker1 # [ 0.159898] iommu: DMA domain TLB invalidation policy: strict mode lb # [ 0.018233] thermal_sys: Registered thermal governor 'power_allocator' worker1 # [ 0.160234] SCSI subsystem initialized worker1 # [ 0.162316] usbcore: registered new interface driver usbfs worker1 # [ 0.162356] usbcore: registered new interface driver hub worker1 # [ 0.162370] usbcore: registered new device driver usb worker1 # [ 0.162657] pps_core: LinuxPPS API ver. 1 registered worker2 # [ 0.178399] VFS: Disk quotas dquot_6.6.0 worker2 # [ 0.178462] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) worker1 # [ 0.162663] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker1 # [ 0.162674] PTP clock support registered worker2 # [ 0.180188] netfs: FS-Cache loaded worker1 # [ 0.162722] EDAC MC: Ver: 3.0.0 worker2 # [ 0.180318] pnp: PnP ACPI: disabled client # [ 0.161163] iommu: Default domain type: Translated worker1 # [ 0.167512] scmi_core: SCMI protocol bus registered client # [ 0.161173] iommu: DMA domain TLB invalidation policy: strict mode worker1 # [ 0.168510] FPGA manager framework client # [ 0.161549] SCSI subsystem initialized worker1 # [ 0.169519] vgaarb: loaded client # [ 0.163594] usbcore: registered new interface driver usbfs worker1 # [ 0.170217] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.018257] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 client # [ 0.163628] usbcore: registered new interface driver hub lb # [ 0.018265] cpuidle: using governor ladder lb # [ 0.018271] cpuidle: using governor menu client # [ 0.163644] usbcore: registered new device driver usb client # [ 0.163976] pps_core: LinuxPPS API ver. 1 registered lb # [ 0.018451] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. lb # [ 0.018468] ASID allocator initialised with 65536 entries client # [ 0.163982] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti lb # [ 0.019714] Serial: AMBA PL011 UART driver client # [ 0.163992] PTP clock support registered worker1 # [ 0.174188] VFS: Disk quotas dquot_6.6.0 client # [ 0.164041] EDAC MC: Ver: 3.0.0 worker2 # [ 0.184376] NET: Registered PF_INET protocol family lb # [ 0.025035] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 lb # [ 0.025161] printk: console [ttyAMA0] enabled worker2 # [ 0.184674] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) client # [ 0.168771] scmi_core: SCMI protocol bus registered lb # [ 0.145747] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages client # [ 0.169923] FPGA manager framework client # [ 0.170985] vgaarb: loaded lb # [ 0.145765] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker1 # [ 0.176833] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) client # [ 0.171608] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.145770] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages worker1 # [ 0.178068] netfs: FS-Cache loaded worker1 # [ 0.178210] pnp: PnP ACPI: disabled lb # [ 0.145774] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page lb # [ 0.145779] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages lb # [ 0.145783] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.145787] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages lb # [ 0.145791] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page client # [ 0.176251] VFS: Disk quotas dquot_6.6.0 client # [ 0.176299] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) lb # [ 0.153216] fbcon: Taking over console lb # [ 0.153229] ACPI: Interpreter disabled. lb # [ 0.155117] iommu: Default domain type: Translated lb # [ 0.155127] iommu: DMA domain TLB invalidation policy: strict mode worker1 # [ 0.182310] NET: Registered PF_INET protocol family client # [ 0.180222] netfs: FS-Cache loaded client # [ 0.180360] pnp: PnP ACPI: disabled lb # [ 0.156798] SCSI subsystem initialized worker1 # [ 0.182631] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) client # [ 0.184435] NET: Registered PF_INET protocol family client # [ 0.184740] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) lb # [ 0.164530] usbcore: registered new interface driver usbfs lb # [ 0.164565] usbcore: registered new interface driver hub lb # [ 0.164590] usbcore: registered new device driver usb lb # [ 0.164878] pps_core: LinuxPPS API ver. 1 registered lb # [ 0.164884] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti lb # [ 0.164895] PTP clock support registered lb # [ 0.164951] EDAC MC: Ver: 3.0.0 lb # [ 0.169650] scmi_core: SCMI protocol bus registered lb # [ 0.170718] FPGA manager framework lb # [ 0.171714] vgaarb: loaded worker2 # [ 0.215273] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) lb # [ 0.172377] clocksource: Switched to clocksource arch_sys_counter worker2 # [ 0.215330] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) lb # [ 0.173032] VFS: Disk quotas dquot_6.6.0 worker2 # [ 0.215362] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) lb # [ 0.173059] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) worker2 # [ 0.215447] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) lb # [ 0.175440] netfs: FS-Cache loaded lb # [ 0.175556] pnp: PnP ACPI: disabled worker2 # [ 0.215589] TCP: Hash tables configured (established 16384 bind 16384) worker2 # [ 0.215684] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker2 # [ 0.221510] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.181713] NET: Registered PF_INET protocol family worker2 # [ 0.221552] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.182009] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker2 # [ 0.221650] NET: Registered PF_UNIX/PF_LOCAL protocol family worker2 # [ 0.221676] NET: Registered PF_XDP protocol family worker2 # [ 0.221698] PCI: CLS 0 bytes, default 64 worker2 # [ 0.221980] Trying to unpack rootfs image as initramfs... worker1 # [ 0.213302] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) worker1 # [ 0.213361] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker1 # [ 0.213394] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) worker1 # [ 0.213476] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker1 # [ 0.213618] TCP: Hash tables configured (established 16384 bind 16384) worker1 # [ 0.213712] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker1 # [ 0.213749] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.213788] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.213886] NET: Registered PF_UNIX/PF_LOCAL protocol family client # [ 0.216079] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) worker1 # [ 0.213907] NET: Registered PF_XDP protocol family worker1 # [ 0.213926] PCI: CLS 0 bytes, default 64 client # [ 0.216135] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker1 # [ 0.214193] Trying to unpack rootfs image as initramfs... client # [ 0.216170] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) client # [ 0.216255] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) client # [ 0.216400] TCP: Hash tables configured (established 16384 bind 16384) worker2 # [ 0.239757] kvm [1]: HYP mode not available client # [ 0.216496] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) client # [ 0.216535] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.216573] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.216672] NET: Registered PF_UNIX/PF_LOCAL protocol family client # [ 0.216725] NET: Registered PF_XDP protocol family worker1 # [ 0.232178] kvm [1]: HYP mode not available client # [ 0.216745] PCI: CLS 0 bytes, default 64 client # [ 0.217010] Trying to unpack rootfs image as initramfs... client # [ 0.233629] kvm [1]: HYP mode not available lb # [ 0.212555] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) lb # [ 0.212615] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) lb # [ 0.212647] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) lb # [ 0.212732] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) lb # [ 0.212877] TCP: Hash tables configured (established 16384 bind 16384) lb # [ 0.212979] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) lb # [ 0.213017] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.213054] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.213155] NET: Registered PF_UNIX/PF_LOCAL protocol family lb # [ 0.213175] NET: Registered PF_XDP protocol family lb # [ 0.213193] PCI: CLS 0 bytes, default 64 lb # [ 0.213461] Trying to unpack rootfs image as initramfs... lb # [ 0.230250] kvm [1]: HYP mode not available worker2 # [ 0.355246] Initialise system trusted keyrings worker2 # [ 0.356096] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker2 # [ 0.357357] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker2 # [ 0.358106] 9p: Installing v9fs 9p2000 file system support worker1 # [ 0.354794] Initialise system trusted keyrings worker1 # [ 0.355552] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker1 # [ 0.356942] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker1 # [ 0.357741] 9p: Installing v9fs 9p2000 file system support client # [ 0.356192] Initialise system trusted keyrings client # [ 0.356976] workingset: timestamp_bits=42 max_order=19 bucket_order=0 client # [ 0.358312] squashfs: version 4.0 (2009/01/31) Phillip Lougher client # [ 0.359120] 9p: Installing v9fs 9p2000 file system support worker2 # [ 0.382250] Key type asymmetric registered worker2 # [ 0.382282] Asymmetric key parser 'x509' registered worker2 # [ 0.382360] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) lb # [ 0.343258] Initialise system trusted keyrings worker2 # [ 0.387799] io scheduler mq-deadline registered lb # [ 0.344026] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker2 # [ 0.387821] io scheduler kyber registered lb # [ 0.345429] squashfs: version 4.0 (2009/01/31) Phillip Lougher lb # [ 0.346194] 9p: Installing v9fs 9p2000 file system support worker1 # [ 0.386562] Key type asymmetric registered worker1 # [ 0.386591] Asymmetric key parser 'x509' registered worker2 # [ 0.399913] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker1 # [ 0.386684] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker1 # [ 0.388887] io scheduler mq-deadline registered worker1 # [ 0.388899] io scheduler kyber registered client # [ 0.387950] Key type asymmetric registered worker2 # [ 0.401776] ledtrig-cpu: registered to indicate activity on CPUs client # [ 0.387981] Asymmetric key parser 'x509' registered worker2 # [ 0.402188] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: client # [ 0.388078] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker2 # [ 0.402205] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 client # [ 0.390254] io scheduler mq-deadline registered client # [ 0.390265] io scheduler kyber registered worker2 # [ 0.402219] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker2 # [ 0.402228] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker1 # [ 0.398384] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker2 # [ 0.402268] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits lb # [ 0.370300] Key type asymmetric registered worker2 # [ 0.402295] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] lb # [ 0.370333] Asymmetric key parser 'x509' registered worker2 # [ 0.402370] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 lb # [ 0.370413] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker2 # [ 0.402380] pci_bus 0000:00: root bus resource [bus 00-ff] worker1 # [ 0.399838] ledtrig-cpu: registered to indicate activity on CPUs lb # [ 0.372574] io scheduler mq-deadline registered worker2 # [ 0.402386] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] lb # [ 0.372584] io scheduler kyber registered worker1 # [ 0.400250] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker2 # [ 0.402391] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.400268] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker2 # [ 0.402396] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.400282] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker2 # [ 0.402473] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.400290] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker2 # [ 0.402932] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.399766] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker2 # [ 0.403126] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.400310] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker2 # [ 0.403144] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.400338] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker2 # [ 0.403174] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.400441] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker2 # [ 0.403191] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.400450] pci_bus 0000:00: root bus resource [bus 00-ff] worker2 # [ 0.403667] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.400457] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker2 # [ 0.403885] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.400462] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker2 # [ 0.403902] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.400466] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker2 # [ 0.403932] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.400560] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint client # [ 0.401297] ledtrig-cpu: registered to indicate activity on CPUs worker2 # [ 0.404386] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker2 # [ 0.404572] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] client # [ 0.401725] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker1 # [ 0.401034] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.404588] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.401222] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] client # [ 0.401747] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker2 # [ 0.404618] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.401239] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.401760] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker1 # [ 0.401269] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.405080] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.405268] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.401286] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.401770] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker2 # [ 0.405284] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.401748] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.401793] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker2 # [ 0.405313] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.401933] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.405332] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.401950] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.401823] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker1 # [ 0.401979] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.405794] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint client # [ 0.401904] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 client # [ 0.401913] pci_bus 0000:00: root bus resource [bus 00-ff] worker2 # [ 0.405983] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.401920] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker2 # [ 0.406013] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.421717] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint client # [ 0.401926] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.421911] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] worker2 # [ 0.406485] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker1 # [ 0.421927] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.401931] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker2 # [ 0.406677] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.421958] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.402019] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker2 # [ 0.406706] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.407148] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint client # [ 0.402464] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.388595] pl061_gpio 9030000.pl061: PL061 GPIO chip registered client # [ 0.402651] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.407332] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] lb # [ 0.389320] ledtrig-cpu: registered to indicate activity on CPUs client # [ 0.402668] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.407580] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.402698] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.389713] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: client # [ 0.402714] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.389730] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 client # [ 0.403196] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.389743] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 client # [ 0.403383] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] client # [ 0.403399] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.389751] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 client # [ 0.403428] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.389778] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits client # [ 0.403914] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint lb # [ 0.389805] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] client # [ 0.404099] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] lb # [ 0.389901] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 client # [ 0.404115] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.389909] pci_bus 0000:00: root bus resource [bus 00-ff] client # [ 0.404145] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.389915] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] client # [ 0.404611] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.389921] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] client # [ 0.404796] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.389926] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] client # [ 0.404811] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.447882] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.430587] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.390047] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint client # [ 0.404841] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.447931] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.430803] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] client # [ 0.404860] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.390509] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.430822] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.448470] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.390695] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] client # [ 0.405327] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker1 # [ 0.430853] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.448672] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.390730] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.405516] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.430873] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.448704] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.390761] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.405546] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.431343] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker2 # [ 0.449174] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.390777] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.406000] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker1 # [ 0.431534] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.449366] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.391265] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.406189] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.449397] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.431565] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.391458] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] client # [ 0.406219] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.391473] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.449865] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker1 # [ 0.432028] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint client # [ 0.406621] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker2 # [ 0.450188] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker1 # [ 0.432220] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.391503] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.406803] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker2 # [ 0.450206] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.432250] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.391957] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint client # [ 0.407120] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.450236] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.392143] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] worker1 # [ 0.432651] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint client # [ 0.407310] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.392159] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.432836] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker2 # [ 0.450734] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.407340] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.450923] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.392188] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.433151] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.450939] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.433343] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.392661] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.450970] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.392844] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.433374] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.392859] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.451571] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker1 # [ 0.433834] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.392888] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.451582] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker1 # [ 0.434024] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.392907] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.451588] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker1 # [ 0.434055] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.393365] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker2 # [ 0.451635] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker1 # [ 0.434531] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.393560] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.434723] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.451684] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned lb # [ 0.393590] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.434754] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.394042] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker1 # [ 0.435235] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint lb # [ 0.394228] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.435554] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] lb # [ 0.394257] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.447958] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.435571] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.469118] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned lb # [ 0.394641] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker1 # [ 0.435602] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.448180] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.394830] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker2 # [ 0.469173] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned client # [ 0.448212] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.436074] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker2 # [ 0.469224] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker1 # [ 0.436262] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.395113] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.448706] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.395300] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.436278] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.448897] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.469274] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned lb # [ 0.395329] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.436309] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.448927] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.469322] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker1 # [ 0.436944] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned lb # [ 0.395780] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.449391] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker2 # [ 0.469370] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned lb # [ 0.395965] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.436955] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned client # [ 0.449704] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker2 # [ 0.469417] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned lb # [ 0.395995] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.449722] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.436961] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker2 # [ 0.469513] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned client # [ 0.449752] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.437008] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker2 # [ 0.469580] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned client # [ 0.450218] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.437056] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker2 # [ 0.469602] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned client # [ 0.450405] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.437104] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned client # [ 0.450421] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.469625] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned client # [ 0.450458] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.469646] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned worker1 # [ 0.437151] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned client # [ 0.451066] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker2 # [ 0.469668] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker1 # [ 0.437198] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker2 # [ 0.469691] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned client # [ 0.451079] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker1 # [ 0.437245] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker2 # [ 0.469713] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned client # [ 0.451085] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker1 # [ 0.437292] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker2 # [ 0.469742] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned client # [ 0.451135] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker1 # [ 0.437338] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker2 # [ 0.469764] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned client # [ 0.451183] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker1 # [ 0.437385] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned worker2 # [ 0.469787] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned client # [ 0.451231] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker2 # [ 0.469809] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker1 # [ 0.437469] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned worker2 # [ 0.469832] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned client # [ 0.451280] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker1 # [ 0.437522] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker2 # [ 0.469854] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned worker1 # [ 0.437544] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned client # [ 0.451328] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker2 # [ 0.469875] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned worker1 # [ 0.437565] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned client # [ 0.451376] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker2 # [ 0.469897] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned worker1 # [ 0.437586] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned lb # [ 0.440542] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.469919] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned client # [ 0.451424] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker1 # [ 0.437608] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned lb # [ 0.440759] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.469942] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned client # [ 0.451470] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker1 # [ 0.437630] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned lb # [ 0.440790] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.469964] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned client # [ 0.451517] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned worker1 # [ 0.437655] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned worker2 # [ 0.469992] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.441335] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker1 # [ 0.437681] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned worker2 # [ 0.470002] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] lb # [ 0.441656] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] client # [ 0.451606] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned lb # [ 0.441673] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.437703] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker2 # [ 0.470006] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] worker2 # [ 0.470889] pci 0000:00:07.0: enabling device (0000 -> 0002) lb # [ 0.441704] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.437725] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned worker1 # [ 0.437748] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned lb # [ 0.442183] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.442366] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.437771] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned lb # [ 0.442381] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.437792] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned worker1 # [ 0.437814] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned lb # [ 0.442410] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.437836] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned lb # [ 0.443015] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker1 # [ 0.437857] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned lb # [ 0.443027] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker1 # [ 0.437879] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned lb # [ 0.443033] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker1 # [ 0.437901] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned worker1 # [ 0.437928] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.443084] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned client # [ 0.479752] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker1 # [ 0.437938] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] lb # [ 0.443131] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned client # [ 0.479780] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker1 # [ 0.437942] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] lb # [ 0.443179] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned client # [ 0.479803] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned lb # [ 0.443227] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned client # [ 0.479827] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned lb # [ 0.443275] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned client # [ 0.479850] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned lb # [ 0.443322] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned client # [ 0.479874] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned client # [ 0.479897] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned lb # [ 0.443370] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned client # [ 0.479924] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned lb # [ 0.443420] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned client # [ 0.479948] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned lb # [ 0.443466] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned client # [ 0.479971] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned lb # [ 0.443573] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned client # [ 0.479995] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned lb # [ 0.443629] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned client # [ 0.480018] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned lb # [ 0.443651] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned client # [ 0.480039] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned lb # [ 0.443672] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned client # [ 0.480062] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned lb # [ 0.443693] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned client # [ 0.480084] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned lb # [ 0.443715] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned client # [ 0.480105] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned client # [ 0.480127] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned lb # [ 0.443736] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned client # [ 0.480149] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned lb # [ 0.443758] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned client # [ 0.480183] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.443784] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned client # [ 0.480193] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] lb # [ 0.443806] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker1 # [ 0.502948] pci 0000:00:07.0: enabling device (0000 -> 0002) client # [ 0.480197] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] lb # [ 0.443827] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned client # [ 0.481034] pci 0000:00:07.0: enabling device (0000 -> 0002) lb # [ 0.443850] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned lb # [ 0.443872] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned lb # [ 0.443893] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned lb # [ 0.443914] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned lb # [ 0.443935] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned lb # [ 0.443956] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned worker2 # [ 0.523599] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) lb # [ 0.443977] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned lb # [ 0.443998] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned lb # [ 0.444024] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.444034] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] lb # [ 0.444038] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] lb # [ 0.444895] pci 0000:00:07.0: enabling device (0000 -> 0002) worker2 # [ 0.533190] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker1 # [ 0.524509] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) client # [ 0.526784] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker2 # [ 0.544413] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.535802] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker2 # [ 0.546625] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) worker2 # [ 0.548815] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) worker2 # [ 0.551125] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) lb # [ 0.511108] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker1 # [ 0.539825] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.542127] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) client # [ 0.537133] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) lb # [ 0.514258] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker2 # [ 0.561285] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker2 # [ 0.563652] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) client # [ 0.542011] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.551935] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) client # [ 0.544241] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) client # [ 0.546425] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) worker2 # [ 0.574889] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) lb # [ 0.524653] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) lb # [ 0.526813] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) worker1 # [ 0.558480] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) lb # [ 0.529041] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) worker1 # [ 0.560666] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) client # [ 0.556574] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) lb # [ 0.531376] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) client # [ 0.558796] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker2 # [ 0.581181] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) lb # [ 0.541654] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) lb # [ 0.544121] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) client # [ 0.570044] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker2 # [ 0.587817] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker1 # [ 0.574992] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker1 # [ 0.577108] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) worker1 # [ 0.580535] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) lb # [ 0.555229] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) worker1 # [ 0.583640] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker2 # [ 0.605031] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled client # [ 0.579872] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) client # [ 0.583685] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) client # [ 0.586550] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) lb # [ 0.561553] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) lb # [ 0.563968] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker2 # [ 0.607772] msm_serial: driver initialized worker2 # [ 0.607951] SuperH (H)SCI(F) driver initialized worker2 # [ 0.608006] STM32 USART driver initialized worker1 # [ 0.603536] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker1 # [ 0.606438] msm_serial: driver initialized worker1 # [ 0.606575] SuperH (H)SCI(F) driver initialized worker1 # [ 0.606628] STM32 USART driver initialized lb # [ 0.580756] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled client # [ 0.604963] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled lb # [ 0.583314] msm_serial: driver initialized lb # [ 0.583461] SuperH (H)SCI(F) driver initialized lb # [ 0.583515] STM32 USART driver initialized client # [ 0.607776] msm_serial: driver initialized client # [ 0.607917] SuperH (H)SCI(F) driver initialized client # [ 0.607972] STM32 USART driver initialized worker2 # [ 0.639045] loop: module loaded worker2 # [ 0.639254] virtio_blk virtio2: 1/0/0 default/read/poll queues worker2 # [ 0.641367] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker1 # [ 0.639178] loop: module loaded worker1 # [ 0.639374] virtio_blk virtio2: 1/0/0 default/read/poll queues worker1 # [ 0.640225] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker2 # [ 0.652458] megasas: 07.734.00.00-rc1 worker2 # [ 0.653213] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] worker2 # [ 0.655658] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.655690] Intel/Sharp Extended Query Table at 0x0031 client # [ 0.642553] loop: module loaded worker2 # [ 0.657430] Using buffer write method worker1 # [ 0.646970] megasas: 07.734.00.00-rc1 client # [ 0.642769] virtio_blk virtio2: 1/0/0 default/read/poll queues worker2 # [ 0.657521] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker1 # [ 0.647705] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] client # [ 0.643586] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker1 # [ 0.650167] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.621021] loop: module loaded worker2 # [ 0.659428] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.621222] virtio_blk virtio2: 1/0/0 default/read/poll queues worker2 # [ 0.659454] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.622029] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker1 # [ 0.658305] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.677855] Using buffer write method worker1 # [ 0.660106] Using buffer write method worker2 # [ 0.677901] Concatenating MTD devices: worker2 # [ 0.677906] (0): "0.flash" worker2 # [ 0.677910] (1): "0.flash" worker1 # [ 0.660192] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker2 # [ 0.677919] into device "0.flash" lb # [ 0.629087] megasas: 07.734.00.00-rc1 worker1 # [ 0.662157] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.662185] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.629782] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] worker1 # [ 0.663984] Using buffer write method worker1 # [ 0.664011] Concatenating MTD devices: worker1 # [ 0.664015] (0): "0.flash" lb # [ 0.632158] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.664018] (1): "0.flash" lb # [ 0.632187] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.664030] into device "0.flash" client # [ 0.656365] megasas: 07.734.00.00-rc1 lb # [ 0.633821] Using buffer write method client # [ 0.657120] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] lb # [ 0.633899] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] client # [ 0.659371] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 client # [ 0.659404] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.635808] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 client # [ 0.661132] Using buffer write method lb # [ 0.635836] Intel/Sharp Extended Query Table at 0x0031 client # [ 0.661202] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] client # [ 0.663038] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 client # [ 0.663093] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.654658] Using buffer write method lb # [ 0.654724] Concatenating MTD devices: lb # [ 0.654728] (0): "0.flash" lb # [ 0.654733] (1): "0.flash" lb # [ 0.654741] into device "0.flash" client # [ 0.681790] Using buffer write method client # [ 0.681839] Concatenating MTD devices: client # [ 0.681843] (0): "0.flash" client # [ 0.681847] (1): "0.flash" client # [ 0.681857] into device "0.flash" worker2 # [ 0.966055] Freeing initrd memory: 26944K worker2 # [ 0.972206] tun: Universal TUN/TAP device driver, 1.6 worker1 # [ 0.964398] Freeing initrd memory: 26976K worker2 # [ 0.976079] thunder_xcv, ver 1.0 worker2 # [ 0.976122] thunder_bgx, ver 1.0 worker2 # [ 0.976150] nicpf, ver 1.0 worker2 # [ 0.976709] e1000: Intel(R) PRO/1000 Network Driver worker2 # [ 0.976719] e1000: Copyright (c) 1999-2006 Intel Corporation. worker2 # [ 0.976741] e1000e: Intel(R) PRO/1000 Network Driver worker2 # [ 0.976750] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker1 # [ 0.970632] tun: Universal TUN/TAP device driver, 1.6 worker2 # [ 0.976778] igb: Intel(R) Gigabit Ethernet Network Driver worker2 # [ 0.976784] igb: Copyright (c) 2007-2014 Intel Corporation. worker2 # [ 0.976806] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker2 # [ 0.976812] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker2 # [ 0.976947] sky2: driver version 1.30 worker2 # [ 0.978922] usbcore: registered new interface driver usb-storage worker2 # [ 0.979054] usbcore: registered new interface driver usbserial_generic worker2 # [ 0.979069] usbserial: USB Serial support registered for generic worker2 # [ 0.979679] hv_vmbus: registering driver hyperv_keyboard worker2 # [ 0.980575] ehci-pci 0000:00:07.0: EHCI Host Controller worker2 # [ 0.980603] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker2 # [ 0.980904] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker1 # [ 0.974568] thunder_xcv, ver 1.0 worker1 # [ 0.974608] thunder_bgx, ver 1.0 worker1 # [ 0.974630] nicpf, ver 1.0 worker1 # [ 0.975200] e1000: Intel(R) PRO/1000 Network Driver worker2 # [ 0.993386] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 client # [ 0.973178] Freeing initrd memory: 26972K worker2 # [ 0.993754] hub 1-0:1.0: USB hub found worker1 # [ 0.975211] e1000: Copyright (c) 1999-2006 Intel Corporation. worker2 # [ 0.993780] hub 1-0:1.0: 6 ports detected worker1 # [ 0.975235] e1000e: Intel(R) PRO/1000 Network Driver worker1 # [ 0.975244] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker1 # [ 0.975274] igb: Intel(R) Gigabit Ethernet Network Driver worker1 # [ 0.975280] igb: Copyright (c) 2007-2014 Intel Corporation. worker2 # [ 0.996974] rtc-pl031 9010000.pl031: registered as rtc0 worker1 # [ 0.975304] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker1 # [ 0.975312] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker2 # [ 0.997016] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T04:42:22 UTC (1790138542) client # [ 0.979533] tun: Universal TUN/TAP device driver, 1.6 worker1 # [ 0.975446] sky2: driver version 1.30 lb # [ 0.955588] Freeing initrd memory: 26948K worker2 # [ 0.997355] i2c_dev: i2c /dev entries driver worker1 # [ 0.977130] usbcore: registered new interface driver usb-storage worker1 # [ 0.977188] usbcore: registered new interface driver usbserial_generic worker1 # [ 0.977201] usbserial: USB Serial support registered for generic worker1 # [ 0.977813] hv_vmbus: registering driver hyperv_keyboard worker1 # [ 0.979027] ehci-pci 0000:00:07.0: EHCI Host Controller client # [ 0.983434] thunder_xcv, ver 1.0 client # [ 0.983482] thunder_bgx, ver 1.0 client # [ 0.983506] nicpf, ver 1.0 worker1 # [ 0.979057] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker2 # [ 1.002931] sdhci: Secure Digital Host Controller Interface driver worker2 # [ 1.002952] sdhci: Copyright(c) Pierre Ossman worker1 # [ 0.979366] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker2 # [ 1.003222] Synopsys Designware Multimedia Card Interface Driver lb # [ 0.961708] tun: Universal TUN/TAP device driver, 1.6 worker2 # [ 1.003598] sdhci-pltfm: SDHCI platform and OF driver helper worker1 # [ 0.991671] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker1 # [ 0.992037] hub 1-0:1.0: USB hub found worker1 # [ 0.992058] hub 1-0:1.0: 6 ports detected client # [ 0.985379] e1000: Intel(R) PRO/1000 Network Driver client # [ 0.985388] e1000: Copyright (c) 1999-2006 Intel Corporation. worker2 # [ 1.008004] hid: raw HID events driver (C) Jiri Kosina worker1 # [ 0.994596] rtc-pl031 9010000.pl031: registered as rtc0 client # [ 0.985417] e1000e: Intel(R) PRO/1000 Network Driver worker2 # [ 1.008263] usbcore: registered new interface driver usbhid worker2 # [ 1.008270] usbhid: USB HID core driver worker1 # [ 0.994625] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T04:42:22 UTC (1790138542) client # [ 0.985424] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker1 # [ 0.995048] i2c_dev: i2c /dev entries driver client # [ 0.985453] igb: Intel(R) Gigabit Ethernet Network Driver client # [ 0.985459] igb: Copyright (c) 2007-2014 Intel Corporation. worker2 # [ 1.011244] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available client # [ 0.985487] igbvf: Intel(R) Gigabit Virtual Function Network Driver lb # [ 0.965642] thunder_xcv, ver 1.0 client # [ 0.985493] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. lb # [ 0.965688] thunder_bgx, ver 1.0 client # [ 0.985634] sky2: driver version 1.30 lb # [ 0.965714] nicpf, ver 1.0 worker2 # [ 1.014031] drop_monitor: Initializing network drop monitor service lb # [ 0.966262] e1000: Intel(R) PRO/1000 Network Driver client # [ 0.987337] usbcore: registered new interface driver usb-storage worker2 # [ 1.014191] NET: Registered PF_INET6 protocol family lb # [ 0.966269] e1000: Copyright (c) 1999-2006 Intel Corporation. client # [ 0.987510] usbcore: registered new interface driver usbserial_generic lb # [ 0.966292] e1000e: Intel(R) PRO/1000 Network Driver worker1 # [ 1.000618] sdhci: Secure Digital Host Controller Interface driver client # [ 0.987526] usbserial: USB Serial support registered for generic worker1 # [ 1.000634] sdhci: Copyright(c) Pierre Ossman worker2 # [ 1.016441] Segment Routing with IPv6 lb # [ 0.966301] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. client # [ 0.988355] ehci-pci 0000:00:07.0: EHCI Host Controller worker2 # [ 1.016466] In-situ OAM (IOAM) with IPv6 worker1 # [ 1.000920] Synopsys Designware Multimedia Card Interface Driver lb # [ 0.966331] igb: Intel(R) Gigabit Ethernet Network Driver worker2 # [ 1.016500] NET: Registered PF_PACKET protocol family client # [ 0.988386] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker1 # [ 1.001306] sdhci-pltfm: SDHCI platform and OF driver helper lb # [ 0.966337] igb: Copyright (c) 2007-2014 Intel Corporation. client # [ 0.988770] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 lb # [ 0.966359] igbvf: Intel(R) Gigabit Virtual Function Network Driver lb # [ 0.966365] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker2 # [ 1.018094] 9pnet: Installing 9P2000 support worker1 # [ 1.005522] hid: raw HID events driver (C) Jiri Kosina lb # [ 0.966497] sky2: driver version 1.30 worker2 # [ 1.018156] Key type dns_resolver registered worker1 # [ 1.005754] usbcore: registered new interface driver usbhid client # [ 1.000041] hv_vmbus: registering driver hyperv_keyboard lb # [ 0.968124] usbcore: registered new interface driver usb-storage worker1 # [ 1.005761] usbhid: USB HID core driver client # [ 1.001503] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 lb # [ 0.968225] usbcore: registered new interface driver usbserial_generic lb # [ 0.968243] usbserial: USB Serial support registered for generic client # [ 1.002614] hub 1-0:1.0: USB hub found client # [ 1.003139] hub 1-0:1.0: 6 ports detected lb # [ 0.969355] ehci-pci 0000:00:07.0: EHCI Host Controller lb # [ 0.969390] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 lb # [ 0.969570] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker1 # [ 1.008709] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available client # [ 1.004311] rtc-pl031 9010000.pl031: registered as rtc0 worker1 # [ 1.011423] drop_monitor: Initializing network drop monitor service lb # [ 0.981221] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker1 # [ 1.011564] NET: Registered PF_INET6 protocol family client # [ 1.004344] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T04:42:22 UTC (1790138542) lb # [ 0.981633] hub 1-0:1.0: USB hub found client # [ 1.004703] i2c_dev: i2c /dev entries driver lb # [ 0.981659] hub 1-0:1.0: 6 ports detected worker1 # [ 1.013538] Segment Routing with IPv6 worker1 # [ 1.013558] In-situ OAM (IOAM) with IPv6 worker2 # [ 1.025147] registered taskstats version 1 worker1 # [ 1.013590] NET: Registered PF_PACKET protocol family worker2 # [ 1.025314] Loading compiled-in X.509 certificates lb # [ 0.984565] hv_vmbus: registering driver hyperv_keyboard lb # [ 0.986183] rtc-pl031 9010000.pl031: registered as rtc0 client # [ 1.010258] sdhci: Secure Digital Host Controller Interface driver worker1 # [ 1.015485] 9pnet: Installing 9P2000 support client # [ 1.010273] sdhci: Copyright(c) Pierre Ossman lb # [ 0.986217] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T04:42:22 UTC (1790138542) worker1 # [ 1.015532] Key type dns_resolver registered client # [ 1.010540] Synopsys Designware Multimedia Card Interface Driver lb # [ 0.986538] i2c_dev: i2c /dev entries driver client # [ 1.010954] sdhci-pltfm: SDHCI platform and OF driver helper worker2 # [ 1.034159] Demotion targets for Node 0: null worker2 # [ 1.034291] Key type .fscrypt registered client # [ 1.015339] hid: raw HID events driver (C) Jiri Kosina worker1 # [ 1.022194] registered taskstats version 1 worker2 # [ 1.034297] Key type fscrypt-provisioning registered lb # [ 0.992143] sdhci: Secure Digital Host Controller Interface driver worker2 # [ 1.034402] ima: No TPM chip found, activating TPM-bypass! lb # [ 0.992157] sdhci: Copyright(c) Pierre Ossman worker2 # [ 1.034421] ima: Allocated hash algorithm: sha1 client # [ 1.016285] usbcore: registered new interface driver usbhid worker2 # [ 1.034463] ima: No architecture policies found client # [ 1.016297] usbhid: USB HID core driver worker2 # [ 1.038666] input: gpio-keys as /devices/platform/gpio-keys/input/input0 client # [ 1.018611] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available lb # [ 0.993743] Synopsys Designware Multimedia Card Interface Driver worker1 # [ 1.022873] Loading compiled-in X.509 certificates lb # [ 0.994134] sdhci-pltfm: SDHCI platform and OF driver helper lb # [ 0.995735] hid: raw HID events driver (C) Jiri Kosina client # [ 1.021420] drop_monitor: Initializing network drop monitor service client # [ 1.021598] NET: Registered PF_INET6 protocol family lb # [ 0.995987] usbcore: registered new interface driver usbhid lb # [ 0.995994] usbhid: USB HID core driver client # [ 1.023765] Segment Routing with IPv6 client # [ 1.023787] In-situ OAM (IOAM) with IPv6 client # [ 1.023839] NET: Registered PF_PACKET protocol family worker1 # [ 1.031292] Demotion targets for Node 0: null worker1 # [ 1.031421] Key type .fscrypt registered worker1 # [ 1.031428] Key type fscrypt-provisioning registered lb # [ 1.000553] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker1 # [ 1.031536] ima: No TPM chip found, activating TPM-bypass! client # [ 1.025508] 9pnet: Installing 9P2000 support lb # [ 1.002133] drop_monitor: Initializing network drop monitor service worker1 # [ 1.031555] ima: Allocated hash algorithm: sha1 client # [ 1.025566] Key type dns_resolver registered lb # [ 1.002302] NET: Registered PF_INET6 protocol family worker1 # [ 1.031576] ima: No architecture policies found lb # [ 1.005710] Segment Routing with IPv6 worker1 # [ 1.035731] input: gpio-keys as /devices/platform/gpio-keys/input/input0 lb # [ 1.005736] In-situ OAM (IOAM) with IPv6 lb # [ 1.005765] NET: Registered PF_PACKET protocol family lb # [ 1.007538] 9pnet: Installing 9P2000 support lb # [ 1.007598] Key type dns_resolver registered client # [ 1.033060] registered taskstats version 1 client # [ 1.033226] Loading compiled-in X.509 certificates worker2 # [ 1.056811] clk: Disabling unused clocks worker2 # [ 1.056854] PM: genpd: Disabling unused power domains lb # [ 1.014591] registered taskstats version 1 lb # [ 1.014762] Loading compiled-in X.509 certificates worker2 # [ 1.061105] Freeing unused kernel memory: 4736K worker2 # [ 1.061279] Run /init as init process client # [ 1.042600] Demotion targets for Node 0: null client # [ 1.042749] Key type .fscrypt registered client # [ 1.042757] Key type fscrypt-provisioning registered client # [ 1.042904] ima: No TPM chip found, activating TPM-bypass! client # [ 1.042934] ima: Allocated hash algorithm: sha1 client # [ 1.042964] ima: No architecture policies found client # [ 1.047246] input: gpio-keys as /devices/platform/gpio-keys/input/input0 lb # [ 1.023610] Demotion targets for Node 0: null lb # [ 1.023755] Key type .fscrypt registered lb # [ 1.023762] Key type fscrypt-provisioning registered lb # [ 1.023890] ima: No TPM chip found, activating TPM-bypass! lb # [ 1.023910] ima: Allocated hash algorithm: sha1 worker1 # [ 1.054655] clk: Disabling unused clocks lb # [ 1.023937] ima: No architecture policies found worker1 # [ 1.054688] PM: genpd: Disabling unused power domains lb # [ 1.028266] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker1 # [ 1.058914] Freeing unused kernel memory: 4736K worker1 # [ 1.059101] Run /init as init process worker2 # [ 1.078141] systemd[1]: Successfully made /usr/ read-only. client # [ 1.066462] clk: Disabling unused clocks client # [ 1.066499] PM: genpd: Disabling unused power domains worker1 # [ 1.076953] systemd[1]: Successfully made /usr/ read-only. client # [ 1.070786] Freeing unused kernel memory: 4736K client # [ 1.071009] Run /init as init process lb # [ 1.047881] clk: Disabling unused clocks lb # [ 1.047921] PM: genpd: Disabling unused power domains lb # [ 1.052172] Freeing unused kernel memory: 4736K lb # [ 1.052980] Run /init as init process client # [ 1.086262] systemd[1]: Successfully made /usr/ read-only. lb # [ 1.071051] systemd[1]: Successfully made /usr/ read-only. worker2 # [ 1.239792] usb 1-1: new high-speed USB device number 2 using ehci-pci worker1 # [ 1.238330] usb 1-1: new high-speed USB device number 2 using ehci-pci client # [ 1.251709] usb 1-1: new high-speed USB device number 2 using ehci-pci lb # [ 1.228467] usb 1-1: new high-speed USB device number 2 using ehci-pci worker2 # [ 1.389640] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker1 # [ 1.392693] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 lb # [ 1.378148] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker2 # [ 1.413274] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 1.425589] systemd[1]: Detected virtualization qemu. worker2 # [ 1.427780] systemd[1]: Detected architecture arm64. worker2 # [ 1.429700] systemd[1]: Running in initrd. client # [ 1.406310] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker2 # [ 1.432505] systemd[1]: Initializing machine ID from random generator. worker2 # [ 1.435580] systemd[1]: Hostname set to . worker1 # [ 1.411949] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker1 # [ 1.424167] systemd[1]: Detected virtualization qemu. worker1 # [ 1.426306] systemd[1]: Detected architecture arm64. worker1 # [ 1.428213] systemd[1]: Running in initrd. worker1 # [ 1.431032] systemd[1]: Initializing machine ID from random generator. worker1 # [ 1.434112] systemd[1]: Hostname set to . client # [ 1.421369] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 1.434941] systemd[1]: Detected virtualization qemu. lb # [ 1.406072] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 1.437144] systemd[1]: Detected architecture arm64. lb # [ 1.414519] systemd[1]: Detected virtualization qemu. lb # [ 1.415704] systemd[1]: Detected architecture arm64. client # [ 1.439121] systemd[1]: Running in initrd. lb # [ 1.416753] systemd[1]: Running in initrd. lb # [ 1.418195] systemd[1]: Initializing machine ID from random generator. client # [ 1.442164] systemd[1]: Initializing machine ID from random generator. lb # [ 1.419891] systemd[1]: Hostname set to . client # [ 1.445350] systemd[1]: Hostname set to . worker2 # [ 1.480069] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker1 # [ 1.482645] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 client # [ 1.492019] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 lb # [ 1.468709] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker2 # [ 1.603819] usb 1-2: new high-speed USB device number 3 using ehci-pci worker1 # [ 1.606262] usb 1-2: new high-speed USB device number 3 using ehci-pci client # [ 1.615649] usb 1-2: new high-speed USB device number 3 using ehci-pci lb # [ 1.592448] usb 1-2: new high-speed USB device number 3 using ehci-pci worker2 # [ 1.726219] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 1.729473] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 1.757866] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.759670] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 client # [ 1.755148] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 1.767968] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker1 # [ 1.770526] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 lb # [ 1.741889] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 1.790883] systemd[1]: Queued start job for default target Initrd Default Target. worker2 # [ 1.801797] systemd[1]: Created slice Slice /system/modprobe. client # [ 1.778148] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.802819] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 1.754984] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.803897] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker2 # [ 1.803932] systemd[1]: Reached target Path Units. worker2 # [ 1.803950] systemd[1]: Reached target Slice Units. client # [ 1.784160] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker2 # [ 1.803967] systemd[1]: Reached target Swaps. worker2 # [ 1.803985] systemd[1]: Reached target Timer Units. worker2 # [ 1.804161] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 1.804289] systemd[1]: Listening on Journal Socket (/dev/log). lb # [ 1.762195] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker2 # [ 1.804409] systemd[1]: Listening on Journal Sockets. worker2 # [ 1.804518] systemd[1]: Listening on udev Control Socket. worker2 # [ 1.804610] systemd[1]: Listening on udev Kernel Socket. worker2 # [ 1.804630] systemd[1]: Reached target Socket Units. worker2 # [ 1.813446] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 1.814368] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 1.808773] systemd[1]: Queued start job for default target Initrd Default Target. worker2 # [ 1.823992] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 1.832043] systemd[1]: Starting Journal Service... worker1 # [ 1.820368] systemd[1]: Created slice Slice /system/modprobe. worker1 # [ 1.821453] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 1.822644] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker1 # [ 1.822679] systemd[1]: Reached target Path Units. worker1 # [ 1.822696] systemd[1]: Reached target Slice Units. worker1 # [ 1.822714] systemd[1]: Reached target Swaps. worker1 # [ 1.822731] systemd[1]: Reached target Timer Units. worker1 # [ 1.822906] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 1.823046] systemd[1]: Listening on Journal Socket (/dev/log). worker1 # [ 1.823177] systemd[1]: Listening on Journal Sockets. worker1 # [ 1.823291] systemd[1]: Listening on udev Control Socket. worker1 # [ 1.823386] systemd[1]: Listening on udev Kernel Socket. worker1 # [ 1.823405] systemd[1]: Reached target Socket Units. worker1 # [ 1.832984] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 1.834078] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 1.842395] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 1.855955] systemd[1]: Starting Load Kernel Modules... worker2 # [ 1.856106] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 1.850549] systemd[1]: Starting Journal Service... worker2 # [ 1.867427] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 1.881690] systemd-journald[72]: Collecting audit messages is disabled. client # [ 1.866348] systemd[1]: Queued start job for default target Initrd Default Target. worker2 # [ 1.884769] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 1.885593] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 1.873876] systemd[1]: Starting Load Kernel Modules... worker1 # [ 1.874003] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 1.873799] systemd[1]: Created slice Slice /system/modprobe. client # [ 1.874880] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 1.876045] systemd[1]: Expecting device /dev/disk/by-label/nixos... client # [ 1.876952] systemd[1]: Reached target Path Units. client # [ 1.877644] systemd[1]: Reached target Slice Units. client # [ 1.878345] systemd[1]: Reached target Swaps. lb # [ 1.854167] systemd[1]: Queued start job for default target Initrd Default Target. client # [ 1.878992] systemd[1]: Reached target Timer Units. client # [ 1.879898] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 1.880938] systemd[1]: Listening on Journal Socket (/dev/log). client # [ 1.881898] systemd[1]: Listening on Journal Sockets. client # [ 1.882764] systemd[1]: Listening on udev Control Socket. client # [ 1.883721] systemd[1]: Listening on udev Kernel Socket. worker2 # [ 1.900124] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 1.884479] systemd[1]: Reached target Socket Units. client # [ 1.886823] systemd[1]: Starting Create List of Static Device Nodes... lb # [ 1.863500] systemd[1]: Created slice Slice /system/modprobe. lb # [ 1.864966] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 1.887866] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 1.894526] systemd[1]: Starting Coldplug All udev Devices... lb # [ 1.866380] systemd[1]: Expecting device /dev/disk/by-label/nixos... lb # [ 1.867526] systemd[1]: Reached target Path Units. worker1 # [ 1.898086] systemd-journald[72]: Collecting audit messages is disabled. lb # [ 1.868433] systemd[1]: Reached target Slice Units. lb # [ 1.869305] systemd[1]: Reached target Swaps. lb # [ 1.870086] systemd[1]: Reached target Timer Units. lb # [ 1.871312] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 1.895796] systemd[1]: Mounting Kernel Configuration File System... lb # [ 1.872667] systemd[1]: Listening on Journal Socket (/dev/log). lb # [ 1.873850] systemd[1]: Listening on Journal Sockets. lb # [ 1.875107] systemd[1]: Listening on udev Control Socket. lb # [ 1.876194] systemd[1]: Listening on udev Kernel Socket. lb # [ 1.877212] systemd[1]: Reached target Socket Units. client # [ 1.903916] systemd[1]: Starting Journal Service... lb # [ 1.880208] systemd[1]: Starting Create List of Static Device Nodes... lb # [ 1.881540] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 1.912495] systemd[1]: Finished Create List of Static Device Nodes. worker1 # [ 1.913236] systemd[1]: Mounted Kernel Configuration File System. lb # [ 1.890082] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 1.923000] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 1.940432] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 1.899015] systemd[1]: Starting Journal Service... worker2 # [ 1.944139] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 1.954986] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. client # [ 1.936016] systemd[1]: Starting Load Kernel Modules... client # [ 1.936157] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 1.961298] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev lb # [ 1.920877] systemd[1]: Starting Load Kernel Modules... lb # [ 1.921856] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 1.930568] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 1.960624] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker2 # [ 1.977188] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker2 # [ 1.977484] [drm] features: -virgl +edid -resource_blob -host_visible worker2 # [ 1.977495] [drm] features: -context_init worker2 # [ 1.978234] [drm] number of scanouts: 1 worker2 # [ 1.978252] [drm] number of cap sets: 0 client # [ 1.961956] systemd-journald[72]: Collecting audit messages is disabled. client # [ 1.963505] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 1.995335] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker2 # [ 1.995368] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 client # [ 1.981369] systemd[1]: Finished Create List of Static Device Nodes. client # [ 1.982112] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 1.986362] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker1 # [ 1.992770] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker1 # [ 1.993068] [drm] features: -virgl +edid -resource_blob -host_visible worker1 # [ 1.993079] [drm] features: -context_init worker1 # [ 1.993791] [drm] number of scanouts: 1 worker1 # [ 1.993809] [drm] number of cap sets: 0 lb # [ 1.965278] systemd[1]: Finished Create List of Static Device Nodes. client # [ 1.992314] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 2.015959] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 2.016437] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 2.016481] systemd[1]: Reached target Local File Systems. worker1 # [ 2.006960] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 2.010655] systemd[1]: Starting Create Static Device Nodes in /dev... client # [ 2.006713] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker2 # [ 2.024260] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker2 # [ 2.033066] Console: switching to colour frame buffer device 160x50 lb # [ 1.998085] systemd-journald[72]: Collecting audit messages is disabled. worker1 # [ 2.022656] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker1 # [ 2.022681] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 client # [ 2.019700] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev client # [ 2.025993] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 lb # [ 2.004658] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 2.026244] [drm] features: -virgl +edid -resource_blob -host_visible lb # [ 2.005383] systemd[1]: Mounted Kernel Configuration File System. client # [ 2.026254] [drm] features: -context_init client # [ 2.027030] [drm] number of scanouts: 1 client # [ 2.027048] [drm] number of cap sets: 0 worker2 # [ 2.056365] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device lb # [ 2.016741] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. lb # [ 2.018556] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker1 # [ 2.042711] Console: switching to colour frame buffer device 160x50 worker2 # [ 2.072450] systemd[1]: Finished Load Kernel Modules. worker2 # [ 2.077090] systemd[1]: Starting Apply Kernel Variables... lb # [ 2.033910] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 lb # [ 2.034174] [drm] features: -virgl +edid -resource_blob -host_visible lb # [ 2.034188] [drm] features: -context_init lb # [ 2.034995] [drm] number of scanouts: 1 lb # [ 2.035019] [drm] number of cap sets: 0 client # [ 2.055850] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic client # [ 2.055875] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 client # [ 2.065807] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 2.075300] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device client # [ 2.068541] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 2.076341] systemd-modules-load[73]: Inserted module 'dm_mod' worker2 # [ 2.077468] systemd-modules-load[73]: Module 'virtio_balloon' is built in worker1 # [ 2.082944] systemd[1]: Finished Create Static Device Nodes in /dev. worker1 # [ 2.083350] systemd[1]: Reached target Preparation for Local File Systems. worker1 # [ 2.083387] systemd[1]: Reached target Local File Systems. lb # [ 2.057121] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 2.088338] systemd-modules-load[73]: Module 'virtio_console' is built in lb # [ 2.060834] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 2.090643] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 2.080022] Console: switching to colour frame buffer device 160x50 worker1 # [ 2.093043] systemd[1]: Started Journal Service. worker2 # [ 2.105278] systemd[1]: Started Journal Service. worker2 # [ 2.093979] systemd-modules-load[73]: Inserted module 'virtio_gpu' worker1 # [ 2.084223] systemd-modules-load[74]: Inserted module 'dm_mod' lb # [ 2.064884] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic lb # [ 2.064911] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 worker1 # [ 2.088286] systemd-modules-load[74]: Module 'virtio_balloon' is built in worker1 # [ 2.089454] systemd-modules-load[74]: Module 'virtio_console' is built in worker1 # [ 2.090543] systemd-modules-load[74]: Inserted module 'virtio_gpu' worker1 # [ 2.091577] systemd-modules-load[74]: Module 'virtio_rng' is built in worker2 # [ 2.103840] systemd-modules-load[73]: Module 'virtio_rng' is built in client # [ 2.104263] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker1 # [ 2.098620] systemd[1]: Finished Load Kernel Modules. worker2 # [ 2.118404] systemd[1]: Starting Create System Files and Directories... worker2 # [ 2.126259] systemd[1]: Finished Apply Kernel Variables. client # [ 2.120390] systemd[1]: Finished Load Kernel Modules. client # [ 2.124005] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 2.117560] systemd[1]: Starting Apply Kernel Variables... lb # [ 2.097060] Console: switching to colour frame buffer device 160x50 worker2 # [ 2.130855] systemd-udevd[79]: Using default interface naming scheme 'v261'. lb # [ 2.101300] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker1 # [ 2.126606] systemd[1]: Starting Create System Files and Directories... lb # [ 2.117688] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 2.117949] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 2.117987] systemd[1]: Reached target Local File Systems. lb # [ 2.129095] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 2.152038] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 2.152525] systemd[1]: Reached target Preparation for Local File Systems. client # [ 2.152569] systemd[1]: Reached target Local File Systems. worker2 # [ 2.161767] systemd[1]: Finished Create System Files and Directories. client # [ 2.158147] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 2.152467] systemd-modules-load[73]: Inserted module 'dm_mod' client # [ 2.153859] systemd-modules-load[73]: Module 'virtio_balloon' is built in worker2 # [ 2.175864] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 2.172397] systemd[1]: Started Journal Service. lb # [ 2.148583] systemd[1]: Finished Load Kernel Modules. client # [ 2.160434] systemd-modules-load[73]: Module 'virtio_console' is built in client # [ 2.165176] systemd-modules-load[73]: Inserted module 'virtio_gpu' lb # [ 2.152556] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 2.173447] systemd[1]: Finished Apply Kernel Variables. client # [ 2.169489] systemd-modules-load[73]: Module 'virtio_rng' is built in client # [ 2.178698] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 2.191521] systemd-udevd[78]: Using default interface naming scheme 'v261'. worker1 # [ 2.194198] systemd[1]: Finished Create System Files and Directories. lb # [ 2.168571] systemd-modules-load[73]: Inserted module 'dm_mod' lb # [ 2.169794] systemd-modules-load[73]: Module 'virtio_balloon' is built in lb # [ 2.170999] systemd-modules-load[73]: Module 'virtio_console' is built in client # [ 2.196178] systemd[1]: Starting Create System Files and Directories... lb # [ 2.192813] systemd[1]: Started Journal Service. lb # [ 2.180946] systemd-modules-load[73]: Inserted module 'virtio_gpu' lb # [ 2.186240] systemd-modules-load[73]: Module 'virtio_rng' is built in worker1 # [ 2.224758] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 2.240533] systemd[1]: Starting Virtual Console Setup... lb # [ 2.209413] systemd[1]: Starting Create System Files and Directories... client # [ 2.237752] systemd[1]: Finished Create System Files and Directories. lb # [ 2.217612] systemd[1]: Finished Apply Kernel Variables. client # [ 2.249672] systemd-udevd[79]: Using default interface naming scheme 'v261'. worker1 # [ 2.284829] systemd[1]: Starting Virtual Console Setup... lb # [ 2.255602] systemd-udevd[78]: Using default interface naming scheme 'v261'. worker2 # [ 2.300595] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 2.281748] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 2.304147] systemd[1]: Finished Virtual Console Setup. lb # [ 2.273252] systemd[1]: Finished Create System Files and Directories. lb # [ 2.297616] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 2.348592] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 2.343320] systemd[1]: Starting Virtual Console Setup... worker1 # [ 2.352276] systemd[1]: Finished Virtual Console Setup. lb # [ 2.360824] systemd[1]: Starting Virtual Console Setup... client # [ 2.404545] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 2.408168] systemd[1]: Finished Virtual Console Setup. lb # [ 2.420558] systemd-vconsole-setup[100]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 2.424469] systemd[1]: Finished Virtual Console Setup. worker2 # [ 2.926426] systemd[1]: Finished Coldplug All udev Devices. worker2 # [ 2.927471] systemd[1]: Reached target System Initialization. worker2 # [ 2.928613] systemd[1]: Reached target Basic System. worker1 # [ 2.971740] systemd[1]: Finished Coldplug All udev Devices. worker1 # [ 2.972842] systemd[1]: Reached target System Initialization. worker1 # [ 2.973692] systemd[1]: Reached target Basic System. client # [ 3.039275] systemd[1]: Finished Coldplug All udev Devices. client # [ 3.044807] systemd[1]: Reached target System Initialization. client # [ 3.045767] systemd[1]: Reached target Basic System. worker2 # [ 3.092107] (udev-worker)[97]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 3.107891] (udev-worker)[101]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 3.122464] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line. lb # [ 3.079589] systemd[1]: Finished Coldplug All udev Devices. lb # [ 3.084810] systemd[1]: Reached target System Initialization. lb # [ 3.085802] systemd[1]: Reached target Basic System. worker1 # [ 3.151960] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.161817] (udev-worker)[99]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 3.168296] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 3.215833] systemd[1]: Found device /dev/disk/by-label/nixos. worker2 # [ 3.218791] systemd[1]: Reached target Initrd Root Device. worker2 # [ 3.223377] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... lb # [ 3.200342] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line. client # [ 3.227010] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 3.269465] systemd-fsck[111]: nixos: clean, 12/65536 files, 13019/262144 blocks worker2 # [ 3.275631] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker2 # [ 3.279942] systemd[1]: Mounting /sysroot... worker1 # [ 3.268975] systemd[1]: Found device /dev/disk/by-label/nixos. worker1 # [ 3.273231] systemd[1]: Reached target Initrd Root Device. client # [ 3.265809] (udev-worker)[100]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 3.275858] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... client # [ 3.269958] (udev-worker)[100]: Network interface NamePolicy= disabled on kernel command line. lb # [ 3.273400] (udev-worker)[99]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. lb # [ 3.278405] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 3.339095] EXT4-fs (vda): mounted filesystem 37680bca-f3e2-43ad-91ac-3b964e5ce3f5 r/w with ordered data mode. Quota mode: none. worker2 # [ 3.330066] systemd[1]: Mounted /sysroot. worker2 # [ 3.333020] systemd[1]: Reached target Initrd Root File System. worker1 # [ 3.325537] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks worker2 # [ 3.339663] systemd[1]: Starting Mountpoints Configured in the Real Root... worker1 # [ 3.333550] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker1 # [ 3.340303] systemd[1]: Mounting /sysroot... worker2 # [ 3.374372] systemd-sysroot-fstab-check[119]: /sysroot should be mounted in the initrd, will request daemon-reload. client # [ 3.356406] systemd[1]: Found device /dev/disk/by-label/nixos. lb # [ 3.336160] systemd[1]: Found device /dev/disk/by-label/nixos. worker2 # [ 3.382232] systemd[1]: Reload requested from client PID 119 ('systemd-sysroot') (unit initrd-parse-etc.service)... client # [ 3.364148] systemd[1]: Reached target Initrd Root Device. lb # [ 3.340769] systemd[1]: Reached target Initrd Root Device. worker2 # [ 3.385615] systemd[1]: Reloading... client # [ 3.368783] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... lb # [ 3.344980] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker1 # [ 3.397174] EXT4-fs (vda): mounted filesystem c161b43a-11e9-4d85-b10c-52abfa81c9ee r/w with ordered data mode. Quota mode: none. worker1 # [ 3.389047] systemd[1]: Mounted /sysroot. worker1 # [ 3.391926] systemd[1]: Reached target Initrd Root File System. worker1 # [ 3.400152] systemd[1]: Starting Mountpoints Configured in the Real Root... worker1 # [ 3.432497] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. client # [ 3.428703] systemd-fsck[111]: nixos: clean, 12/65536 files, 13019/262144 blocks worker1 # [ 3.438671] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... lb # [ 3.409749] systemd-fsck[109]: nixos: clean, 12/65536 files, 13019/262144 blocks worker1 # [ 3.443912] systemd[1]: Reloading... client # [ 3.440237] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. lb # [ 3.416189] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. client # [ 3.443303] systemd[1]: Mounting /sysroot... lb # [ 3.421006] systemd[1]: Mounting /sysroot... lb # [ 3.475415] EXT4-fs (vda): mounted filesystem 8622face-eab8-4b8c-b722-453650c9cee3 r/w with ordered data mode. Quota mode: none. lb # [ 3.465061] systemd[1]: Mounted /sysroot. lb # [ 3.467692] systemd[1]: Reached target Initrd Root File System. client # [ 3.507418] EXT4-fs (vda): mounted filesystem 9b461f11-b1d6-4956-868d-e04030333ae3 r/w with ordered data mode. Quota mode: none. client # [ 3.499717] systemd[1]: Mounted /sysroot. lb # [ 3.476141] systemd[1]: Starting Mountpoints Configured in the Real Root... client # [ 3.504110] systemd[1]: Reached target Initrd Root File System. client # [ 3.510403] systemd[1]: Starting Mountpoints Configured in the Real Root... lb # [ 3.495937] systemd-sysroot-fstab-check[117]: /sysroot should be mounted in the initrd, will request daemon-reload. lb # [ 3.503481] systemd[1]: Reload requested from client PID 117 ('systemd-sysroot') (unit initrd-parse-etc.service)... lb # [ 3.509577] systemd[1]: Reloading... client # [ 3.545599] systemd-sysroot-fstab-check[119]: /sysroot should be mounted in the initrd, will request daemon-reload. client # [ 3.554477] systemd[1]: Reload requested from client PID 119 ('systemd-sysroot') (unit initrd-parse-etc.service)... client # [ 3.558029] systemd[1]: Reloading... worker2 # [ 3.629779] systemd[1]: Reloading finished in 246 ms. worker2 # [ 3.662725] systemd-sysroot-fstab-check[119]: Requesting initrd-fs.target/start/replace... worker2 # [ 3.668302] systemd-sysroot-fstab-check[119]: Requesting swap.target/start/replace... worker2 # [ 3.677104] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker2 # [ 3.680644] systemd[1]: Finished Mountpoints Configured in the Real Root. worker2 # [ 3.683054] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker1 # [ 3.686821] systemd[1]: Reloading finished in 245 ms. worker1 # [ 3.716640] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... worker1 # [ 3.722724] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... worker1 # [ 3.730850] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker1 # [ 3.733771] systemd[1]: Finished Mountpoints Configured in the Real Root. worker1 # [ 3.735527] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. lb # [ 3.736141] systemd[1]: Reloading finished in 229 ms. lb # [ 3.764842] systemd-sysroot-fstab-check[117]: Requesting initrd-fs.target/start/replace... lb # [ 3.769595] systemd-sysroot-fstab-check[117]: Requesting swap.target/start/replace... lb # [ 3.775921] systemd[1]: initrd-parse-etc.service: Deactivated successfully. client # [ 3.803018] systemd[1]: Reloading finished in 250 ms. lb # [ 3.779221] systemd[1]: Finished Mountpoints Configured in the Real Root. lb # [ 3.782165] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. client # [ 3.835032] systemd-sysroot-fstab-check[119]: Requesting initrd-fs.target/start/replace... client # [ 3.838451] systemd-sysroot-fstab-check[119]: Requesting swap.target/start/replace... worker2 # [ 3.858277] systemd[1]: Mounting /sysroot/nix/.ro-store... client # [ 3.846169] systemd[1]: initrd-parse-etc.service: Deactivated successfully. client # [ 3.849292] systemd[1]: Finished Mountpoints Configured in the Real Root. client # [ 3.851696] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker2 # [ 3.871659] systemd[1]: Mounting /sysroot/nix/.rw-store... worker2 # [ 3.891453] systemd[1]: Mounting /sysroot/run... worker2 # [ 3.902186] systemd[1]: Mounting /sysroot/tmp/shared... worker2 # [ 3.916588] systemd[1]: Mounting /sysroot/tmp/xchg... worker1 # [ 3.907331] systemd[1]: Mounting /sysroot/nix/.ro-store... worker1 # [ 3.924171] systemd[1]: Mounting /sysroot/nix/.rw-store... worker1 # [ 3.934911] systemd[1]: Mounting /sysroot/run... worker1 # [ 3.946525] systemd[1]: Mounting /sysroot/tmp/shared... lb # [ 3.945324] systemd[1]: Mounting /sysroot/nix/.ro-store... worker1 # [ 3.978862] systemd[1]: Mounting /sysroot/tmp/xchg... client # [ 3.972997] systemd[1]: Mounting /sysroot/nix/.ro-store... worker2 # [ 4.009749] fuse: init (API version 7.45) lb # [ 3.962228] systemd[1]: Mounting /sysroot/nix/.rw-store... client # [ 3.991520] systemd[1]: Mounting /sysroot/nix/.rw-store... worker2 # [ 4.026725] virtiofs virtio6: discovered new tag: nix-store lb # [ 3.969481] systemd[1]: Mounting /sysroot/run... worker2 # [ 4.027643] virtiofs virtio6: virtio_fs_setup_dax: No cache capability worker2 # [ 4.020871] systemd[1]: Mounted /sysroot/nix/.rw-store. client # [ 4.001570] systemd[1]: Mounting /sysroot/run... worker2 # [ 4.023174] systemd[1]: Mounted /sysroot/run. worker2 # [ 4.041668] virtiofs virtio7: discovered new tag: shared worker2 # [ 4.042558] virtiofs virtio7: virtio_fs_setup_dax: No cache capability worker1 # [ 4.018495] systemd[1]: Mounted /sysroot/nix/.rw-store. worker2 # [ 4.050262] virtiofs virtio8: discovered new tag: xchg worker2 # [ 4.051130] virtiofs virtio8: virtio_fs_setup_dax: No cache capability lb # [ 3.995398] systemd[1]: Mounting /sysroot/tmp/shared... client # [ 4.025713] systemd[1]: Mounting /sysroot/tmp/shared... lb # [ 4.005321] systemd[1]: Mounting /sysroot/tmp/xchg... worker1 # [ 4.047915] systemd[1]: Mounted /sysroot/run. worker2 # [ 4.065128] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 4.055261] systemd[1]: Mounting /sysroot/tmp/xchg... worker2 # [ 4.075546] systemd[1]: Mounted /sysroot/nix/.ro-store. worker2 # [ 4.083309] systemd[1]: Mounted /sysroot/tmp/shared. worker1 # [ 4.084770] fuse: init (API version 7.45) worker2 # [ 4.085340] systemd[1]: Mounted /sysroot/tmp/xchg. lb # [ 4.047388] systemd[1]: Mounted /sysroot/nix/.rw-store. worker1 # [ 4.092405] virtiofs virtio6: discovered new tag: nix-store worker1 # [ 4.093307] virtiofs virtio6: virtio_fs_setup_dax: No cache capability worker1 # [ 4.108186] virtiofs virtio7: discovered new tag: shared worker1 # [ 4.109092] virtiofs virtio7: virtio_fs_setup_dax: No cache capability worker2 # [ 4.111074] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 4.114152] systemd[1]: Finished rw-sysroot-nix-store.service. client # [ 4.109630] fuse: init (API version 7.45) worker2 # [ 4.116264] systemd[1]: Mounting /sysroot/nix/store... worker1 # [ 4.116658] virtiofs virtio8: discovered new tag: xchg worker1 # [ 4.117591] virtiofs virtio8: virtio_fs_setup_dax: No cache capability lb # [ 4.078136] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 4.110597] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 4.096777] fuse: init (API version 7.45) client # [ 4.124760] virtiofs virtio6: discovered new tag: nix-store client # [ 4.125632] virtiofs virtio6: virtio_fs_setup_dax: No cache capability lb # [ 4.104560] virtiofs virtio6: discovered new tag: nix-store lb # [ 4.105401] virtiofs virtio6: virtio_fs_setup_dax: No cache capability client # [ 4.115695] systemd[1]: Mounted /sysroot/nix/.rw-store. worker1 # [ 4.128857] systemd[1]: Mounted /sysroot/nix/.ro-store. worker1 # [ 4.132177] systemd[1]: Mounted /sysroot/tmp/shared. client # [ 4.142687] virtiofs virtio7: discovered new tag: shared client # [ 4.143536] virtiofs virtio7: virtio_fs_setup_dax: No cache capability lb # [ 4.123457] virtiofs virtio7: discovered new tag: shared lb # [ 4.124292] virtiofs virtio7: virtio_fs_setup_dax: No cache capability worker1 # [ 4.143112] systemd[1]: Mounted /sysroot/tmp/xchg. client # [ 4.153391] virtiofs virtio8: discovered new tag: xchg client # [ 4.154240] virtiofs virtio8: virtio_fs_setup_dax: No cache capability lb # [ 4.131942] virtiofs virtio8: discovered new tag: xchg lb # [ 4.121399] systemd[1]: Mounted /sysroot/run. lb # [ 4.139744] virtiofs virtio8: virtio_fs_setup_dax: No cache capability worker2 # [ 4.170120] systemd[1]: Mounted /sysroot/nix/store. worker2 # [ 4.172163] systemd[1]: Reached target Initrd File Systems. client # [ 4.154011] systemd[1]: Mounted /sysroot/run. worker2 # [ 4.174582] systemd[1]: Starting Find NixOS closure... worker1 # [ 4.168183] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker1 # [ 4.171691] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.185230] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker1 # [ 4.174398] systemd[1]: Mounting /sysroot/nix/store... lb # [ 4.146643] systemd[1]: Mounted /sysroot/nix/.ro-store. client # [ 4.178188] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 4.180259] systemd[1]: Mounted /sysroot/nix/.ro-store. client # [ 4.182155] systemd[1]: Mounted /sysroot/tmp/shared. lb # [ 4.157230] systemd[1]: Mounted /sysroot/tmp/shared. lb # [ 4.161584] systemd[1]: Mounted /sysroot/tmp/xchg. lb # [ 4.164316] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 4.191342] systemd[1]: Mounted /sysroot/tmp/xchg. lb # [ 4.167811] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.223690] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. client # [ 4.216477] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 4.218018] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.242971] systemd[1]: Finished Find NixOS closure. worker1 # [ 4.231367] systemd[1]: Mounted /sysroot/nix/store. worker1 # [ 4.233963] systemd[1]: Reached target Initrd File Systems. worker2 # [ 4.246745] systemd[1]: Reached target Initrd Default Target. worker2 # [ 4.249240] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker1 # [ 4.240224] systemd[1]: Starting Find NixOS closure... worker1 # [ 4.246308] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker2 # [ 4.283489] systemd[1]: Stopped target Initrd Default Target. worker2 # [ 4.286814] systemd[1]: Stopped target Basic System. worker2 # [ 4.288316] systemd[1]: Stopped target Initrd Root Device. worker2 # [ 4.294037] systemd[1]: Stopped target Path Units. worker2 # [ 4.296097] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker2 # [ 4.297462] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker2 # [ 4.304332] systemd[1]: Stopped target Slice Units. worker2 # [ 4.306426] systemd[1]: Stopped target Socket Units. worker1 # [ 4.294261] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker2 # [ 4.307873] systemd[1]: Stopped target System Initialization. worker2 # [ 4.309593] systemd[1]: Stopped target Swaps. worker2 # [ 4.313562] systemd[1]: Stopped target Timer Units. worker2 # [ 4.315075] systemd[1]: dbus.socket: Deactivated successfully. worker2 # [ 4.319572] systemd[1]: Closed D-Bus System Message Bus Socket. worker2 # [ 4.321805] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker1 # [ 4.310796] systemd[1]: Finished Find NixOS closure. worker2 # [ 4.325525] systemd[1]: Stopped Find NixOS closure. worker1 # [ 4.315287] systemd[1]: Reached target Initrd Default Target. worker2 # [ 4.329728] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 4.319058] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker2 # [ 4.332957] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker2 # [ 4.335517] systemd[1]: Stopped Apply Kernel Variables. worker2 # [ 4.337936] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker2 # [ 4.339727] systemd[1]: Stopped Load Kernel Modules. worker2 # [ 4.350977] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker2 # [ 4.352311] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker2 # [ 4.355585] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker2 # [ 4.360989] systemd[1]: Stopped Create System Files and Directories. worker2 # [ 4.362009] systemd[1]: Stopped target Local File Systems. worker2 # [ 4.362824] systemd[1]: Stopped target Preparation for Local File Systems. worker2 # [ 4.363773] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. worker2 # [ 4.370038] systemd[1]: Stopped Coldplug All udev Devices. worker1 # [ 4.359365] systemd[1]: Stopped target Initrd Default Target. worker2 # [ 4.372161] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker2 # [ 4.374235] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 4.375457] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 4.364382] systemd[1]: Stopped target Basic System. worker2 # [ 4.376558] systemd[1]: initrd-cleanup.service: Deactivated successfully. worker2 # [ 4.377520] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker1 # [ 4.366023] systemd[1]: Stopped target Initrd Root Device. worker2 # [ 4.378449] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 4.379744] systemd[1]: Finished rw-sysroot-nix-store.service. worker1 # [ 4.368119] systemd[1]: Stopped target Path Units. worker1 # [ 4.371149] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker2 # [ 4.385963] systemd[1]: systemd-udevd.service: Deactivated successfully. worker1 # [ 4.373826] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker2 # [ 4.387977] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker1 # [ 4.377984] systemd[1]: Stopped target Slice Units. worker2 # [ 4.392326] systemd[1]: systemd-udevd.service: Consumed 1.066s CPU time over 2.375s wall clock time, 16.4M memory peak. worker2 # [ 4.393850] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. worker2 # [ 4.396484] systemd[1]: Closed udev Control Socket. worker1 # [ 4.384181] systemd[1]: Stopped target Socket Units. worker2 # [ 4.397310] systemd[1]: Starting Cleanup udev Database... worker1 # [ 4.386118] systemd[1]: Stopped target System Initialization. worker2 # [ 4.398401] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker2 # [ 4.399609] systemd[1]: Stopped Create Static Device Nodes in /dev. worker1 # [ 4.388164] systemd[1]: Stopped target Swaps. worker1 # [ 4.388885] systemd[1]: Stopped target Timer Units. worker2 # [ 4.400654] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker2 # [ 4.401797] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. worker1 # [ 4.390690] systemd[1]: dbus.socket: Deactivated successfully. worker2 # [ 4.402803] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker2 # [ 4.403790] systemd[1]: Stopped Create List of Static Device Nodes. worker1 # [ 4.397780] systemd[1]: Closed D-Bus System Message Bus Socket. worker1 # [ 4.402222] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker1 # [ 4.408875] systemd[1]: Stopped Find NixOS closure. worker1 # [ 4.412172] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 4.415086] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker1 # [ 4.418701] systemd[1]: Stopped Apply Kernel Variables. worker1 # [ 4.419551] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker2 # [ 4.433568] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker2 # [ 4.436609] systemd[1]: Finished Cleanup udev Database. worker1 # [ 4.425517] systemd[1]: Stopped Load Kernel Modules. worker1 # [ 4.426297] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker2 # [ 4.438949] systemd[1]: Reached target Switch Root. worker1 # [ 4.427455] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker2 # [ 4.440720] systemd[1]: Starting NixOS Activation... worker1 # [ 4.433464] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker1 # [ 4.436178] systemd[1]: Stopped Create System Files and Directories. worker1 # [ 4.440315] systemd[1]: Stopped target Local File Systems. worker1 # [ 4.441188] systemd[1]: Stopped target Preparation for Local File Systems. worker1 # [ 4.442141] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. worker1 # [ 4.444361] systemd[1]: Stopped Coldplug All udev Devices. worker1 # [ 4.446296] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker1 # [ 4.450050] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.451267] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 4.454674] systemd[1]: initrd-cleanup.service: Deactivated successfully. worker1 # [ 4.455736] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker1 # [ 4.457317] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker1 # [ 4.458426] systemd[1]: Finished rw-sysroot-nix-store.service. worker1 # [ 4.463451] systemd[1]: systemd-udevd.service: Deactivated successfully. worker1 # [ 4.466334] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker1 # [ 4.467716] systemd[1]: systemd-udevd.service: Consumed 1.075s CPU time over 2.386s wall clock time, 16.1M memory peak. worker1 # [ 4.472634] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. worker1 # [ 4.473688] systemd[1]: Closed udev Control Socket. worker1 # [ 4.476316] systemd[1]: Starting Cleanup udev Database... worker1 # [ 4.479024] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker1 # [ 4.480325] systemd[1]: Stopped Create Static Device Nodes in /dev. worker1 # [ 4.481255] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker1 # [ 4.482665] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. worker1 # [ 4.483685] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker1 # [ 4.484802] systemd[1]: Stopped Create List of Static Device Nodes. worker1 # [ 4.515672] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker1 # [ 4.520089] systemd[1]: Finished Cleanup udev Database. worker2 # [ 4.531734] initrd-nixos-activation-start[168]: booting system configuration /nix/store/rcm8j4pnk0zmlmgsymh11jg6bfkli2cx-nixos-system-worker2-test worker1 # [ 4.521414] systemd[1]: Reached target Switch Root. worker1 # [ 4.524789] systemd[1]: Starting NixOS Activation... worker2 # [ 4.568125] initrd-nixos-activation-start[168]: running activation script... worker1 # [ 4.621210] initrd-nixos-activation-start[169]: booting system configuration /nix/store/bn37sc2n1i20ygvgim4ix1wq2wbb4gsw-nixos-system-worker1-test worker1 # [ 4.656139] initrd-nixos-activation-start[169]: running activation script... lb # [ 4.758527] (udev-worker)[93]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory lb # [ 4.765370] (udev-worker)[93]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory worker2 # [ 4.831617] initrd-nixos-activation-start[191]: setting up /etc... lb # [ 4.805955] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.809064] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.810223] systemd[1]: Stopping Virtual Console Setup... lb # [ 4.812281] systemd[1]: Starting Virtual Console Setup... lb # [ 4.842111] systemd-vconsole-setup[151]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 4.845812] systemd[1]: Finished Virtual Console Setup. client # [ 4.888702] (udev-worker)[101]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory client # [ 4.891696] (udev-worker)[101]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory worker1 # [ 4.934360] initrd-nixos-activation-start[192]: setting up /etc... client # [ 4.929778] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.932809] systemd[1]: Stopped Virtual Console Setup. client # [ 4.936320] systemd[1]: Stopping Virtual Console Setup... client # [ 4.937172] systemd[1]: Starting Virtual Console Setup... worker2 # [ 4.966946] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker2 # [ 4.970037] systemd[1]: Finished NixOS Activation. worker2 # [ 4.971486] systemd[1]: Starting Switch Root... client # [ 4.954654] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.956482] systemd[1]: Stopped Virtual Console Setup. client # [ 4.964367] systemd[1]: Starting Virtual Console Setup... lb # [ 4.946750] systemd[1]: Mounting /sysroot/nix/store... client # [ 4.973518] systemd[1]: Mounting /sysroot/nix/store... worker2 # [ 4.996864] systemd[1]: Switching root. client # [ 5.003072] systemd-vconsole-setup[155]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 5.010109] systemd[1]: Finished Virtual Console Setup. client # [ 5.011010] systemd[1]: sysroot-run-credentials-systemd\x2dvconsole\x2dsetup.service.mount: Deactivated successfully. client # [ 5.030529] systemd[1]: Mounted /sysroot/nix/store. client # [ 5.031444] systemd[1]: Reached target Initrd File Systems. lb # [ 5.008138] systemd[1]: Mounted /sysroot/nix/store. client # [ 5.033788] systemd[1]: Starting Find NixOS closure... lb # [ 5.009588] systemd[1]: Reached target Initrd File Systems. lb # [ 5.014152] systemd[1]: Starting Find NixOS closure... client # [ 5.040615] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... lb # [ 5.024453] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker1 # [ 5.064813] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker1 # [ 5.067779] systemd[1]: Finished NixOS Activation. worker1 # [ 5.069216] systemd[1]: Starting Switch Root... client # [ 5.070616] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. client # [ 5.082900] systemd[1]: Finished Find NixOS closure. worker1 # [ 5.093256] systemd[1]: Switching root. client # [ 5.085917] systemd[1]: Reached target Initrd Default Target. client # [ 5.087491] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... lb # [ 5.068227] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. lb # [ 5.072273] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. lb # [ 5.086428] systemd[1]: Finished Find NixOS closure. client # [ 5.113521] systemd[1]: Stopped target Initrd Default Target. lb # [ 5.089617] systemd[1]: Reached target Initrd Default Target. client # [ 5.114870] systemd[1]: Stopped target Basic System. client # [ 5.115832] systemd[1]: Stopped target Initrd Root Device. lb # [ 5.091293] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... client # [ 5.117032] systemd[1]: Stopped target Path Units. client # [ 5.120167] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. client # [ 5.121623] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. client # [ 5.123744] systemd[1]: Stopped target Slice Units. client # [ 5.128174] systemd[1]: Stopped target Socket Units. client # [ 5.129052] systemd[1]: Stopped target System Initialization. client # [ 5.130503] systemd[1]: Stopped target Swaps. client # [ 5.132119] systemd[1]: Stopped target Timer Units. client # [ 5.137631] systemd[1]: dbus.socket: Deactivated successfully. client # [ 5.140531] systemd[1]: Closed D-Bus System Message Bus Socket. worker2 # [ 5.175998] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). lb # [ 5.120833] systemd[1]: Stopped target Initrd Default Target. lb # [ 5.122766] systemd[1]: Stopped target Basic System. client # [ 5.147343] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. client # [ 5.148866] systemd[1]: Stopped Find NixOS closure. lb # [ 5.125152] systemd[1]: Stopped target Initrd Root Device. lb # [ 5.126833] systemd[1]: Stopped target Path Units. lb # [ 5.130855] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. client # [ 5.156656] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 5.132250] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. client # [ 5.159045] systemd[1]: systemd-sysctl.service: Deactivated successfully. lb # [ 5.136150] systemd[1]: Stopped target Slice Units. client # [ 5.161993] systemd[1]: Stopped Apply Kernel Variables. lb # [ 5.137080] systemd[1]: Stopped target Socket Units. lb # [ 5.138222] systemd[1]: Stopped target System Initialization. client # [ 5.165646] systemd[1]: systemd-modules-load.service: Deactivated successfully. lb # [ 5.140559] systemd[1]: Stopped target Swaps. client # [ 5.169137] systemd[1]: Stopped Load Kernel Modules. client # [ 5.170789] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. lb # [ 5.148542] systemd[1]: Stopped target Timer Units. client # [ 5.173116] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. lb # [ 5.149382] systemd[1]: dbus.socket: Deactivated successfully. lb # [ 5.150260] systemd[1]: Closed D-Bus System Message Bus Socket. lb # [ 5.151175] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. client # [ 5.176367] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. client # [ 5.177438] systemd[1]: Stopped Create System Files and Directories. client # [ 5.180415] systemd[1]: Stopped target Local File Systems. client # [ 5.181255] systemd[1]: Stopped target Preparation for Local File Systems. client # [ 5.183320] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. client # [ 5.188309] systemd[1]: Stopped Coldplug All udev Devices. lb # [ 5.164368] systemd[1]: Stopped Find NixOS closure. client # [ 5.189118] systemd[1]: Stopping Rule-based Manager for Device Events and Files... lb # [ 5.165232] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 5.190138] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 5.191162] systemd[1]: Stopped Virtual Console Setup. lb # [ 5.166350] systemd[1]: systemd-sysctl.service: Deactivated successfully. client # [ 5.191917] systemd[1]: systemd-udevd.service: Deactivated successfully. lb # [ 5.167349] systemd[1]: Stopped Apply Kernel Variables. client # [ 5.193630] systemd[1]: Stopped Rule-based Manager for Device Events and Files. client # [ 5.194639] systemd[1]: systemd-udevd.service: Consumed 1.650s CPU time over 3.022s wall clock time, 22.3M memory peak. client # [ 5.196116] systemd[1]: initrd-cleanup.service: Deactivated successfully. client # [ 5.197027] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. client # [ 5.197914] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 5.198872] systemd[1]: Finished rw-sysroot-nix-store.service. client # [ 5.199682] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. lb # [ 5.177926] systemd[1]: systemd-modules-load.service: Deactivated successfully. client # [ 5.204338] systemd[1]: Closed udev Control Socket. client # [ 5.206342] systemd[1]: Starting Cleanup udev Database... lb # [ 5.181582] systemd[1]: Stopped Load Kernel Modules. client # [ 5.208484] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. lb # [ 5.184443] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. lb # [ 5.186668] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. client # [ 5.212219] systemd[1]: Stopped Create Static Device Nodes in /dev. client # [ 5.213156] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. lb # [ 5.190492] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. client # [ 5.216187] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. lb # [ 5.191590] systemd[1]: Stopped Create System Files and Directories. client # [ 5.217240] systemd[1]: kmod-static-nodes.service: Deactivated successfully. lb # [ 5.193139] systemd[1]: Stopped target Local File Systems. client # [ 5.220119] systemd[1]: Stopped Create List of Static Device Nodes. lb # [ 5.196435] systemd[1]: Stopped target Preparation for Local File Systems. lb # [ 5.197492] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. lb # [ 5.199426] systemd[1]: Stopped Coldplug All udev Devices. lb # [ 5.203298] systemd[1]: Stopping Rule-based Manager for Device Events and Files... lb # [ 5.204760] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 5.205789] systemd[1]: Stopped Virtual Console Setup. lb # [ 5.206573] systemd[1]: initrd-cleanup.service: Deactivated successfully. lb # [ 5.207525] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. lb # [ 5.208574] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 5.209577] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 5.210436] systemd[1]: systemd-udevd.service: Deactivated successfully. lb # [ 5.211363] systemd[1]: Stopped Rule-based Manager for Device Events and Files. lb # [ 5.216297] systemd[1]: systemd-udevd.service: Consumed 1.528s CPU time over 3.073s wall clock time, 22.4M memory peak. lb # [ 5.217802] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. lb # [ 5.224509] systemd[1]: Closed udev Control Socket. lb # [ 5.225366] systemd[1]: Starting Cleanup udev Database... client # [ 5.250494] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. lb # [ 5.226160] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. lb # [ 5.227272] systemd[1]: Stopped Create Static Device Nodes in /dev. lb # [ 5.228316] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. client # [ 5.254288] systemd[1]: Finished Cleanup udev Database. client # [ 5.255202] systemd[1]: Reached target Switch Root. client # [ 5.256397] systemd[1]: Starting NixOS Activation... lb # [ 5.232173] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. lb # [ 5.233246] systemd[1]: kmod-static-nodes.service: Deactivated successfully. lb # [ 5.236106] systemd[1]: Stopped Create List of Static Device Nodes. worker1 # [ 5.282341] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). lb # [ 5.265340] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. lb # [ 5.268581] systemd[1]: Finished Cleanup udev Database. lb # [ 5.269448] systemd[1]: Reached target Switch Root. lb # [ 5.270870] systemd[1]: Starting NixOS Activation... client # [ 5.347559] initrd-nixos-activation-start[178]: booting system configuration /nix/store/h0h42468m1lycnnbixi8y4srnpd3qprr-nixos-system-client-test client # [ 5.383591] initrd-nixos-activation-start[178]: running activation script... lb # [ 5.362332] initrd-nixos-activation-start[174]: booting system configuration /nix/store/kgjzg78npi7z07am2szqxkr41f06jp5w-nixos-system-lb-test lb # [ 5.397594] initrd-nixos-activation-start[174]: running activation script... client # [ 5.648314] initrd-nixos-activation-start[201]: setting up /etc... lb # [ 5.655784] initrd-nixos-activation-start[197]: setting up /etc... client # [ 5.793004] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. client # [ 5.794816] systemd[1]: Finished NixOS Activation. client # [ 5.798034] systemd[1]: Starting Switch Root... client # [ 5.818084] systemd[1]: Switching root. lb # [ 5.794162] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. lb # [ 5.797525] systemd[1]: Finished NixOS Activation. lb # [ 5.798927] systemd[1]: Starting Switch Root... worker2 # [ 5.870027] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 5.875498] systemd[1]: Detected virtualization qemu. worker2 # [ 5.878750] systemd[1]: Detected architecture arm64. worker2 # [ 5.882619] systemd[1]: Detected first boot. lb # [ 5.826971] systemd[1]: Switching root. worker1 # [ 5.863300] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 5.889143] systemd[1]: Initializing machine ID from random generator. worker1 # [ 5.876900] systemd[1]: Detected virtualization qemu. worker1 # [ 5.880408] systemd[1]: Detected architecture arm64. worker1 # [ 5.884753] systemd[1]: Detected first boot. worker1 # [ 5.891348] systemd[1]: Initializing machine ID from random generator. client # [ 6.008127] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). lb # [ 6.016348] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker2 # [ 6.225246] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 6.233560] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 6.425911] systemd[1]: Applying preset policy. worker1 # [ 6.541523] systemd[1]: Applying preset policy. lb # [ 6.565722] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 6.578797] systemd[1]: Detected virtualization qemu. lb # [ 6.582231] systemd[1]: Detected architecture arm64. lb # [ 6.586908] systemd[1]: Detected first boot. client # [ 6.601870] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 6.615520] systemd[1]: Detected virtualization qemu. lb # [ 6.594346] systemd[1]: Initializing machine ID from random generator. client # [ 6.619881] systemd[1]: Detected architecture arm64. client # [ 6.623742] systemd[1]: Detected first boot. client # [ 6.630960] systemd[1]: Initializing machine ID from random generator. worker2 # [ 6.750034] systemd[1]: Populated /etc with preset unit settings. worker1 # [ 6.828653] systemd[1]: Populated /etc with preset unit settings. lb # [ 6.990602] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 7.039986] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker2 # [ 7.041442] systemd[1]: Stopped initrd-switch-root.service. worker2 # [ 7.046202] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. client # [ 7.028860] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 7.052091] systemd[1]: Created slice Slice /system/getty. worker2 # [ 7.054917] systemd[1]: Created slice User and Session Slice. worker2 # [ 7.057492] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker2 # [ 7.060300] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker2 # [ 7.062761] systemd[1]: Expecting device /dev/hvc0... worker2 # [ 7.064931] systemd[1]: Expecting device /dev/ttyAMA0... worker2 # [ 7.067295] systemd[1]: Reached target Local Encrypted Volumes. worker2 # [ 7.069536] systemd[1]: Stopped target initrd-fs.target. worker2 # [ 7.071689] systemd[1]: Stopped target initrd-root-fs.target. worker2 # [ 7.073759] systemd[1]: Stopped target initrd-switch-root.target. worker2 # [ 7.075837] systemd[1]: Reached target Virtual Machines and Containers. worker2 # [ 7.077899] systemd[1]: Reached target Path Units. worker2 # [ 7.079802] systemd[1]: Reached target Remote File Systems. worker2 # [ 7.081685] systemd[1]: Reached target Slice Units. worker2 # [ 7.083497] systemd[1]: Reached target Swaps. worker2 # [ 7.087285] systemd[1]: Listening on Query the User Interactively for a Password. worker2 # [ 7.090967] systemd[1]: Listening on Process Core Dump Socket. worker2 # [ 7.093560] systemd[1]: Listening on Credential Encryption/Decryption. worker2 # [ 7.096298] systemd[1]: Listening on Factory Reset Management. worker2 # [ 7.097569] systemd[1]: Listening on Hostname Service Socket. worker2 # [ 7.103322] systemd[1]: Starting Journal Log Access Socket... worker2 # [ 7.106203] systemd[1]: Listening on Journal Audit Socket. worker2 # [ 7.109622] systemd[1]: Listening on Console Output Muting Service Socket. worker2 # [ 7.111425] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker2 # [ 7.113532] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 7.115316] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 7.122802] systemd[1]: Listening on Disk Repartitioning Service Socket. worker2 # [ 7.124416] systemd[1]: Listening on udev Control Socket. worker2 # [ 7.126181] systemd[1]: Listening on udev Varlink Socket. worker1 # [ 7.115950] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker2 # [ 7.130783] systemd[1]: Mounting Huge Pages File System... worker1 # [ 7.118608] systemd[1]: Stopped initrd-switch-root.service. worker1 # [ 7.123415] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker1 # [ 7.128666] systemd[1]: Created slice Slice /system/getty. worker2 # [ 7.139941] systemd[1]: Mounting POSIX Message Queue File System... worker1 # [ 7.131183] systemd[1]: Created slice User and Session Slice. worker1 # [ 7.133523] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 7.136542] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker1 # [ 7.139095] systemd[1]: Expecting device /dev/hvc0... worker1 # [ 7.140159] systemd[1]: Expecting device /dev/ttyAMA0... worker2 # [ 7.152185] systemd[1]: Mounting Kernel Debug File System... worker1 # [ 7.143053] systemd[1]: Reached target Local Encrypted Volumes. worker1 # [ 7.144825] systemd[1]: Stopped target initrd-fs.target. worker1 # [ 7.146472] systemd[1]: Stopped target initrd-root-fs.target. worker1 # [ 7.149021] systemd[1]: Stopped target initrd-switch-root.target. worker1 # [ 7.150209] systemd[1]: Reached target Virtual Machines and Containers. worker2 # [ 7.164656] systemd[1]: Mounting Kernel Trace File System... worker1 # [ 7.152363] systemd[1]: Reached target Path Units. worker1 # [ 7.154934] systemd[1]: Reached target Remote File Systems. worker1 # [ 7.156101] systemd[1]: Reached target Slice Units. worker1 # [ 7.157753] systemd[1]: Reached target Swaps. worker1 # [ 7.161577] systemd[1]: Listening on Query the User Interactively for a Password. worker1 # [ 7.164946] systemd[1]: Listening on Process Core Dump Socket. worker1 # [ 7.167679] systemd[1]: Listening on Credential Encryption/Decryption. worker2 # [ 7.183241] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 7.170469] systemd[1]: Listening on Factory Reset Management. worker1 # [ 7.171797] systemd[1]: Listening on Hostname Service Socket. worker2 # [ 7.187929] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 7.177149] systemd[1]: Starting Journal Log Access Socket... worker1 # [ 7.179930] systemd[1]: Listening on Journal Audit Socket. worker1 # [ 7.184277] systemd[1]: Listening on Console Output Muting Service Socket. worker1 # [ 7.186063] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker1 # [ 7.188647] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.190801] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 7.206446] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 7.208338] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm worker1 # [ 7.197607] systemd[1]: Listening on Disk Repartitioning Service Socket. worker1 # [ 7.199279] systemd[1]: Listening on udev Control Socket. worker1 # [ 7.201097] systemd[1]: Listening on udev Varlink Socket. worker1 # [ 7.205676] systemd[1]: Mounting Huge Pages File System... worker2 # [ 7.216128] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker2 # [ 7.216675] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker1 # [ 7.215099] systemd[1]: Mounting POSIX Message Queue File System... worker2 # [ 7.240491] systemd[1]: Mounting FUSE Control File System... worker1 # [ 7.226619] systemd[1]: Mounting Kernel Debug File System... worker2 # [ 7.241073] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker1 # [ 7.246682] systemd[1]: Mounting Kernel Trace File System... worker1 # [ 7.255431] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 7.257877] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 7.277847] systemd[1]: Starting Journal Service... worker1 # [ 7.278575] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 7.279385] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm worker1 # [ 7.279814] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker1 # [ 7.280281] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 7.258494] systemd[1]: Applying preset policy. worker1 # [ 7.304595] systemd[1]: Mounting FUSE Control File System... worker2 # [ 7.323091] systemd[1]: Starting Load Kernel Modules... worker1 # [ 7.311649] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 client # [ 7.310474] systemd[1]: Applying preset policy. worker2 # [ 7.342762] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker2 # [ 7.352369] systemd[1]: Starting Remount Root and Kernel File Systems... worker2 # [ 7.352866] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.349899] systemd[1]: Starting Journal Service... worker2 # [ 7.364701] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 7.371373] systemd[1]: Listening on Journal Log Access Socket. worker2 # [ 7.377259] systemd[1]: Mounted Huge Pages File System. worker2 # [ 7.380776] systemd[1]: Mounted POSIX Message Queue File System. worker2 # [ 7.384950] systemd[1]: Mounted Kernel Debug File System. worker2 # [ 7.388132] systemd[1]: Mounted Kernel Trace File System. worker2 # [ 7.392294] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 7.394914] systemd[1]: Mounted FUSE Control File System. worker1 # [ 7.393477] systemd[1]: Starting Load Kernel Modules... worker2 # [ 7.422808] systemd[1]: Finished Create List of Static Device Nodes. worker1 # [ 7.411129] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker2 # [ 7.432387] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker1 # [ 7.432227] systemd[1]: Starting Remount Root and Kernel File Systems... worker1 # [ 7.434833] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.442914] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 7.449983] systemd[1]: Listening on Journal Log Access Socket. worker1 # [ 7.459567] systemd[1]: Mounted Huge Pages File System. worker1 # [ 7.462987] systemd[1]: Mounted POSIX Message Queue File System. worker1 # [ 7.466742] systemd[1]: Mounted Kernel Debug File System. worker1 # [ 7.470680] systemd[1]: Mounted Kernel Trace File System. worker1 # [ 7.473482] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 7.477237] systemd[1]: Mounted FUSE Control File System. worker1 # [ 7.505483] systemd[1]: Finished Create List of Static Device Nodes. worker1 # [ 7.520772] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 7.537603] systemd[1]: Finished Load Kernel Modules. worker2 # [ 7.545916] systemd[1]: Starting Firewall... worker2 # [ 7.551219] EXT4-fs (vda): re-mounted 37680bca-f3e2-43ad-91ac-3b964e5ce3f5. worker2 # [ 7.557364] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 7.561659] systemd-journald[262]: Collecting audit messages is enabled. worker2 # [ 7.585160] systemd[1]: Finished Remount Root and Kernel File Systems. worker2 # [ 7.586089] systemd[1]: Listening on Disk Image Download Service Socket. worker2 # [ 7.586538] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 7.598892] systemd[1]: Starting Load/Save OS Random Seed... worker2 # [ 7.603697] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.611023] systemd[1]: Finished Load Kernel Modules. worker1 # [ 7.620339] systemd[1]: Starting Firewall... worker2 # [ 7.638523] systemd[1]: Started Journal Service. worker1 # [ 7.626360] systemd-journald[263]: Collecting audit messages is enabled. worker1 # [ 7.641180] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 7.642699] systemd[1]: Queued start job for default target Multi-User System. worker2 # [ 7.649164] systemd[1]: systemd-journald.service: Deactivated successfully. worker2 # [ 7.665685] systemd-modules-load[263]: Module 'atkbd' is built in worker1 # [ 7.667459] EXT4-fs (vda): re-mounted c161b43a-11e9-4d85-b10c-52abfa81c9ee. worker1 # [ 7.681494] systemd[1]: Started Journal Service. worker2 # [ 7.681912] systemd-modules-load[263]: Module 'loop' is built in worker1 # [ 7.688367] systemd[1]: Queued start job for default target Multi-User System. worker2 # [ 7.692758] systemd[1]: Starting Flush Journal to Persistent Storage... worker1 # [ 7.689725] systemd[1]: systemd-journald.service: Deactivated successfully. worker1 # [ 7.710360] systemd-modules-load[264]: Module 'atkbd' is built in worker1 # [ 7.726647] systemd-modules-load[264]: Module 'loop' is built in worker1 # [ 7.736151] systemd[1]: Finished Remount Root and Kernel File Systems. worker1 # [ 7.753979] systemd[1]: Listening on Disk Image Download Service Socket. worker1 # [ 7.770715] systemd[1]: Starting Flush Journal to Persistent Storage... worker2 # [ 7.790892] systemd-oomd[264]: No swap; memory pressure usage will be degraded worker1 # [ 7.783715] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 7.810762] systemd[1]: Finished Load/Save OS Random Seed. worker1 # [ 7.802572] systemd[1]: Starting Load/Save OS Random Seed... client # [ 7.816178] systemd[1]: Populated /etc with preset unit settings. worker2 # [ 7.824690] systemd[1]: Reached target First Boot Complete. worker1 # [ 7.816686] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 7.834153] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 7.835186] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker2 # [ 7.842793] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 7.852498] systemd[1]: Starting Create Static Device Nodes in /dev... lb # [ 7.832610] systemd[1]: Populated /etc with preset unit settings. worker2 # [ 7.881583] systemd-journald[262]: Received client request to flush runtime journal. worker1 # [ 7.875259] systemd-oomd[265]: No swap; memory pressure usage will be degraded worker1 # [ 7.906721] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker1 # [ 7.930735] systemd-journald[263]: Received client request to flush runtime journal. worker2 # [ 7.952901] systemd[1]: Finished Flush Journal to Persistent Storage. worker1 # [ 8.000574] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 8.007256] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 8.013172] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 8.014177] systemd[1]: Finished Load/Save OS Random Seed. worker1 # [ 8.015016] systemd[1]: Reached target First Boot Complete. worker1 # [ 8.022692] systemd[1]: Finished Flush Journal to Persistent Storage. worker2 # [ 8.100531] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 8.101728] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 8.113346] systemd[1]: Mounting /run/wrappers... worker2 # [ 8.115388] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 8.134928] systemd[1]: initrd-switch-root.service: Deactivated successfully. client # [ 8.137455] systemd[1]: Stopped initrd-switch-root.service. client # [ 8.141910] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. client # [ 8.147441] systemd[1]: Created slice Slice /system/getty. client # [ 8.149951] systemd[1]: Created slice User and Session Slice. client # [ 8.152397] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 8.153927] systemd[1]: Started Forward Password Requests to Wall Directory Watch. client # [ 8.156829] systemd[1]: Expecting device /dev/hvc0... client # [ 8.159218] systemd[1]: Expecting device /dev/ttyAMA0... worker1 # [ 8.154039] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 8.161604] systemd[1]: Reached target Local Encrypted Volumes. worker1 # [ 8.155270] systemd[1]: Reached target Preparation for Local File Systems. client # [ 8.163966] systemd[1]: Stopped target initrd-fs.target. client # [ 8.166058] systemd[1]: Stopped target initrd-root-fs.target. client # [ 8.168291] systemd[1]: Stopped target initrd-switch-root.target. client # [ 8.170673] systemd[1]: Reached target Virtual Machines and Containers. worker1 # [ 8.165103] systemd[1]: Mounting /run/wrappers... client # [ 8.173120] systemd[1]: Reached target Path Units. client # [ 8.175184] systemd[1]: Reached target Remote File Systems. worker1 # [ 8.168118] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 8.177557] systemd[1]: Reached target Slice Units. client # [ 8.179689] systemd[1]: Reached target Swaps. client # [ 8.184018] systemd[1]: Listening on Query the User Interactively for a Password. client # [ 8.187784] systemd[1]: Listening on Process Core Dump Socket. lb # [ 8.163580] systemd[1]: initrd-switch-root.service: Deactivated successfully. client # [ 8.190509] systemd[1]: Listening on Credential Encryption/Decryption. lb # [ 8.165987] systemd[1]: Stopped initrd-switch-root.service. client # [ 8.193521] systemd[1]: Listening on Factory Reset Management. client # [ 8.194938] systemd[1]: Listening on Hostname Service Socket. lb # [ 8.171043] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. client # [ 8.200871] systemd[1]: Starting Journal Log Access Socket... lb # [ 8.177077] systemd[1]: Created slice Slice /system/getty. client # [ 8.203153] systemd[1]: Listening on Journal Audit Socket. lb # [ 8.179466] systemd[1]: Created slice User and Session Slice. client # [ 8.206886] systemd[1]: Listening on Console Output Muting Service Socket. lb # [ 8.181824] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker2 # [ 8.212432] systemd[1]: Mounted /run/wrappers. worker2 # [ 8.213330] systemd[1]: Reached target Local File Systems. client # [ 8.208894] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. lb # [ 8.184329] systemd[1]: Started Forward Password Requests to Wall Directory Watch. lb # [ 8.186667] systemd[1]: Expecting device /dev/hvc0... client # [ 8.211570] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os lb # [ 8.189196] systemd[1]: Expecting device /dev/ttyAMA0... client # [ 8.214995] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki lb # [ 8.191613] systemd[1]: Reached target Local Encrypted Volumes. lb # [ 8.193904] systemd[1]: Stopped target initrd-fs.target. worker2 # [ 8.223595] systemd[1]: Listening on Boot Loader Control Service Socket. lb # [ 8.196717] systemd[1]: Stopped target initrd-root-fs.target. client # [ 8.223686] systemd[1]: Listening on Disk Repartitioning Service Socket. lb # [ 8.198923] systemd[1]: Stopped target initrd-switch-root.target. client # [ 8.225454] systemd[1]: Listening on udev Control Socket. lb # [ 8.201152] systemd[1]: Reached target Virtual Machines and Containers. lb # [ 8.203384] systemd[1]: Reached target Path Units. client # [ 8.227351] systemd[1]: Listening on udev Varlink Socket. worker2 # [ 8.232955] systemd[1]: Starting register-nix-paths.service... lb # [ 8.205412] systemd[1]: Reached target Remote File Systems. lb # [ 8.207651] systemd[1]: Reached target Slice Units. client # [ 8.233069] systemd[1]: Mounting Huge Pages File System... lb # [ 8.209708] systemd[1]: Reached target Swaps. lb # [ 8.214385] systemd[1]: Listening on Query the User Interactively for a Password. worker2 # [ 8.244102] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 8.240413] systemd[1]: Mounting POSIX Message Queue File System... lb # [ 8.218979] systemd[1]: Listening on Process Core Dump Socket. lb # [ 8.222286] systemd[1]: Listening on Credential Encryption/Decryption. lb # [ 8.225269] systemd[1]: Listening on Factory Reset Management. worker2 # [ 8.253952] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. lb # [ 8.226777] systemd[1]: Listening on Hostname Service Socket. worker1 # [ 8.244235] systemd[1]: Mounted /run/wrappers. lb # [ 8.233276] systemd[1]: Starting Journal Log Access Socket... lb # [ 8.235445] systemd[1]: Listening on Journal Audit Socket. client # [ 8.259972] systemd[1]: Mounting Kernel Debug File System... lb # [ 8.239920] systemd[1]: Listening on Console Output Muting Service Socket. lb # [ 8.241903] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker1 # [ 8.261679] systemd[1]: Reached target Local File Systems. lb # [ 8.245937] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 8.262674] systemd[1]: Listening on Boot Loader Control Service Socket. lb # [ 8.248291] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki client # [ 8.272544] systemd[1]: Mounting Kernel Trace File System... worker2 # [ 8.279424] systemd[1]: Starting Save Transient machine-id to Disk... worker1 # [ 8.269224] systemd[1]: Starting register-nix-paths.service... worker2 # [ 8.284261] systemd[1]: Starting Create System Files and Directories... lb # [ 8.256695] systemd[1]: Listening on Disk Repartitioning Service Socket. lb # [ 8.258287] systemd[1]: Listening on udev Control Socket. lb # [ 8.260100] systemd[1]: Listening on udev Varlink Socket. lb # [ 8.265577] systemd[1]: Mounting Huge Pages File System... worker1 # [ 8.288365] systemd[1]: Starting Create SUID/SGID Wrappers... lb # [ 8.272802] systemd[1]: Mounting POSIX Message Queue File System... client # [ 8.295915] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 8.297937] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. client # [ 8.304784] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 8.287110] systemd[1]: Mounting Kernel Debug File System... worker1 # [ 8.312353] systemd[1]: Starting Save Transient machine-id to Disk... worker1 # [ 8.320331] systemd[1]: Starting Create System Files and Directories... lb # [ 8.307262] systemd[1]: Mounting Kernel Trace File System... client # [ 8.344302] systemd[1]: Mounting Kernel Configuration File System... client # [ 8.349770] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm lb # [ 8.325656] systemd[1]: Starting Create List of Static Device Nodes... client # [ 8.354245] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore lb # [ 8.332724] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 8.363114] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 8.360849] systemd[1]: Mounting Kernel Configuration File System... client # [ 8.389034] systemd[1]: Mounting FUSE Control File System... lb # [ 8.365102] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm lb # [ 8.367306] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore lb # [ 8.377073] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse client # [ 8.403510] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 8.414662] systemd[1]: Mounting FUSE Control File System... client # [ 8.442472] systemd[1]: Starting Journal Service... lb # [ 8.421807] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 client # [ 8.461007] systemd[1]: Starting Load Kernel Modules... client # [ 8.479267] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... lb # [ 8.456779] systemd[1]: Starting Journal Service... lb # [ 8.476086] systemd[1]: Starting Load Kernel Modules... client # [ 8.504172] systemd[1]: Starting Remount Root and Kernel File Systems... lb # [ 8.496588] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... client # [ 8.521327] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 8.510178] systemd[1]: Starting Remount Root and Kernel File Systems... lb # [ 8.511812] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 8.551424] systemd[1]: Starting Coldplug All udev Devices... client # [ 8.568866] systemd-journald[271]: Collecting audit messages is enabled. lb # [ 8.546096] systemd-journald[267]: Collecting audit messages is enabled. lb # [ 8.554855] systemd[1]: Starting Coldplug All udev Devices... client # [ 8.579146] systemd[1]: Listening on Journal Log Access Socket. lb # [ 8.560192] systemd[1]: Listening on Journal Log Access Socket. client # [ 8.583542] systemd[1]: Mounted Huge Pages File System. client # [ 8.586573] systemd[1]: Mounted POSIX Message Queue File System. lb # [ 8.568950] systemd[1]: Mounted Huge Pages File System. lb # [ 8.555765] systemd[1]: Queued start job for default target Multi-User System. client # [ 8.595029] systemd[1]: Mounted Kernel Debug File System. lb # [ 8.574066] systemd[1]: Started Journal Service. lb # [ 8.560628] systemd[1]: systemd-journald.service: Deactivated successfully. client # [ 8.600327] systemd[1]: Mounted Kernel Trace File System. lb # [ 8.561808] systemd[1]: Mounted POSIX Message Queue File System. lb # [ 8.562857] systemd[1]: Mounted Kernel Debug File System. lb # [ 8.563783] systemd[1]: Mounted Kernel Trace File System. client # [ 8.593541] systemd[1]: Queued start job for default target Multi-User System. client # [ 8.596629] systemd[1]: systemd-journald.service: Deactivated successfully. client # [ 8.598212] systemd-modules-load[272]: Module 'atkbd' is built in lb # [ 8.573762] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 8.621262] systemd[1]: Finished Create System Files and Directories. lb # [ 8.584222] systemd[1]: Mounted Kernel Configuration File System. client # [ 8.625612] systemd[1]: Started Journal Service. client # [ 8.612506] systemd-modules-load[272]: Module 'loop' is built in worker2 # [ 8.634527] systemd[1]: Starting Rebuild Journal Catalog... worker1 # [ 8.626991] systemd[1]: Finished Create System Files and Directories. client # [ 8.629565] systemd[1]: Finished Create List of Static Device Nodes. client # [ 8.633308] systemd[1]: Finished Load Kernel Modules. lb # [ 8.624554] EXT4-fs (vda): re-mounted 8622face-eab8-4b8c-b722-453650c9cee3. worker1 # [ 8.641820] systemd[1]: Starting Rebuild Journal Catalog... worker2 # [ 8.654139] systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [ 8.638709] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 8.656343] systemd[1]: Starting Record System Boot/Shutdown in UTMP... lb # [ 8.624581] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 8.632581] systemd-modules-load[269]: Module 'atkbd' is built in lb # [ 8.633492] systemd[1]: Finished Remount Root and Kernel File Systems. lb # [ 8.634417] systemd[1]: Listening on Disk Image Download Service Socket. lb # [ 8.635347] systemd-modules-load[269]: Module 'loop' is built in client # [ 8.669297] systemd[1]: Mounted FUSE Control File System. lb # [ 8.645407] systemd[1]: Starting Flush Journal to Persistent Storage... client # [ 8.670602] systemd-oomd[273]: No swap; memory pressure usage will be degraded lb # [ 8.646509] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore client # [ 8.687927] EXT4-fs (vda): re-mounted 9b461f11-b1d6-4956-868d-e04030333ae3. lb # [ 8.659675] systemd[1]: Starting Load/Save OS Random Seed... client # [ 8.687962] systemd[1]: Starting Apply Kernel Variables... client # [ 8.691887] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 8.667852] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 8.694131] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. lb # [ 8.670696] systemd[1]: Finished Load Kernel Modules. client # [ 8.696541] systemd[1]: Finished Remount Root and Kernel File Systems. client # [ 8.714439] systemd[1]: Listening on Disk Image Download Service Socket. client # [ 8.724736] systemd[1]: Starting Flush Journal to Persistent Storage... client # [ 8.725835] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore lb # [ 8.708165] systemd-oomd[270]: No swap; memory pressure usage will be degraded client # [ 8.736112] systemd[1]: Starting Load/Save OS Random Seed... client # [ 8.740814] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 8.717086] systemd[1]: Mounted FUSE Control File System. lb # [ 8.725405] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker2 # [ 8.781210] systemd[1]: Finished Record System Boot/Shutdown in UTMP. lb # [ 8.740193] systemd[1]: Starting Firewall... lb # [ 8.748441] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 8.790507] systemd[1]: Finished Save Transient machine-id to Disk. worker2 # [ 8.802951] systemd[1]: Finished Save Transient machine-id to Disk. worker1 # [ 8.802672] systemd[1]: Finished Record System Boot/Shutdown in UTMP. lb # [ 8.813023] systemd-journald[267]: Received client request to flush runtime journal. client # [ 8.849383] systemd-journald[271]: Received client request to flush runtime journal. worker2 # [ 8.899137] systemd[1]: Finished Rebuild Journal Catalog. lb # [ 8.855327] systemd[1]: Finished Load/Save OS Random Seed. lb # [ 8.864873] systemd[1]: Reached target First Boot Complete. lb # [ 8.865805] systemd[1]: Finished Flush Journal to Persistent Storage. client # [ 8.898626] systemd[1]: Finished Load/Save OS Random Seed. client # [ 8.899859] systemd[1]: Reached target First Boot Complete. worker2 # [ 8.920361] systemd[1]: Starting Update is Completed... worker1 # [ 8.911420] systemd[1]: Finished Rebuild Journal Catalog. worker1 # [ 8.916809] systemd[1]: Starting Update is Completed... client # [ 8.910954] systemd[1]: Finished Flush Journal to Persistent Storage. client # [ 8.917111] systemd[1]: Finished Apply Kernel Variables. lb # [ 8.920291] systemd[1]: Finished Apply Kernel Variables. lb # [ 8.926286] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 8.934357] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 8.995132] systemd-udevd[301]: Using default interface naming scheme 'v261'. worker2 # [ 9.025253] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker1 # [ 9.022356] systemd-udevd[303]: Using default interface naming scheme 'v261'. worker2 # [ 9.034774] systemd[1]: Finished Update is Completed. worker1 # [ 9.048994] systemd[1]: Finished Update is Completed. worker1 # [ 9.102522] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. client # [ 9.110889] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 9.116989] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 9.240617] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 9.228934] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 9.231214] systemd[1]: Reached target Preparation for Local File Systems. worker1 # [ 9.262996] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 9.243420] systemd[1]: Mounting /run/wrappers... lb # [ 9.244712] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 9.328741] systemd[1]: Mounted /run/wrappers. lb # [ 9.329551] systemd[1]: Reached target Local File Systems. lb # [ 9.342508] systemd[1]: Listening on Boot Loader Control Service Socket. lb # [ 9.348237] systemd[1]: Starting register-nix-paths.service... lb # [ 9.355381] systemd[1]: Starting Create SUID/SGID Wrappers... lb # [ 9.356773] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. lb # [ 9.365511] systemd[1]: Starting Save Transient machine-id to Disk... lb # [ 9.409508] systemd[1]: Starting Create System Files and Directories... lb # [ 9.446090] systemd-udevd[304]: Using default interface naming scheme 'v261'. client # [ 9.502313] systemd[1]: Finished Coldplug All udev Devices. client # [ 9.509952] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 9.512483] systemd[1]: Reached target Preparation for Local File Systems. client # [ 9.515953] systemd[1]: Mounting /run/wrappers... client # [ 9.524436] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 9.569738] systemd[1]: Mounted /run/wrappers. client # [ 9.570594] systemd[1]: Reached target Local File Systems. client # [ 9.575293] systemd[1]: Listening on Boot Loader Control Service Socket. client # [ 9.579724] systemd[1]: Starting register-nix-paths.service... client # [ 9.587885] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 9.589412] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. client # [ 9.595901] systemd[1]: Starting Save Transient machine-id to Disk... lb # [ 9.574631] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. client # [ 9.612930] systemd[1]: Starting Create System Files and Directories... worker2 # [ 9.633026] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. lb # [ 9.589217] systemd[1]: Finished Save Transient machine-id to Disk. worker2 # [ 9.634270] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 9.623555] systemd-udevd[302]: Using default interface naming scheme 'v261'. worker1 # [ 9.633041] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker1 # [ 9.642140] systemd[1]: Finished Create SUID/SGID Wrappers. lb # [ 9.623591] systemd[1]: Finished Create System Files and Directories. lb # [ 9.636353] systemd[1]: Starting Rebuild Journal Catalog... lb # [ 9.648498] systemd[1]: Starting Record System Boot/Shutdown in UTMP... lb # [ 9.669136] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 9.719614] systemd[1]: Finished Save Transient machine-id to Disk. client # [ 9.774559] systemd[1]: Finished Create System Files and Directories. client # [ 9.780824] systemd[1]: Starting Rebuild Journal Catalog... client # [ 9.792963] systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [ 9.794064] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 9.803364] systemd[1]: Finished Record System Boot/Shutdown in UTMP. lb # [ 9.844677] systemd[1]: Finished Rebuild Journal Catalog. lb # [ 9.855179] systemd[1]: Starting Update is Completed... client # [ 9.902629] systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 9.924625] systemd[1]: Finished Rebuild Journal Catalog. client # [ 9.933654] systemd[1]: Starting Update is Completed... lb # [ 9.927251] systemd[1]: Finished Update is Completed. client # [ 10.026022] systemd[1]: Finished Update is Completed. client # [ 10.119873] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker2 # [ 10.212116] systemd[1]: Finished Firewall. client # [ 10.203552] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 10.240596] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker1 # [ 10.307111] systemd[1]: Finished Firewall. lb # [ 10.370702] systemd[1]: Finished Coldplug All udev Devices. worker2 # [ 10.448136] systemd[1]: Finished Coldplug All udev Devices. lb # [ 10.438374] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 10.475527] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker1 # [ 10.505829] systemd[1]: Finished register-nix-paths.service. worker2 # [ 10.520851] systemd[1]: Finished register-nix-paths.service. worker2 # [ 10.521872] systemd[1]: Reached target System Initialization. worker2 # [ 10.522746] systemd[1]: Started Discard unused filesystem blocks once a week. worker2 # [ 10.524478] systemd[1]: Started Daily Cleanup of Temporary Directories. worker2 # [ 10.526660] systemd[1]: Reached target Timer Units. worker2 # [ 10.527970] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 10.533740] systemd[1]: Listening on Nix Daemon Socket. worker2 # [ 10.538120] systemd[1]: Listening on Nix worker-protocol over gRPC. client # [ 10.522079] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. client # [ 10.524870] systemd[1]: Finished Create SUID/SGID Wrappers. worker2 # [ 10.547402] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. lb # [ 10.502904] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker2 # [ 10.566438] systemd[1]: Reached target Socket Units. worker2 # [ 10.573321] systemd[1]: Reached target Basic System. client # [ 10.559052] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 10.579155] systemd[1]: Starting Import lastlog data into lastlog2 database... worker2 # [ 10.586214] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 10.594100] systemd[1]: Starting Post-Boot Actions... worker1 # [ 10.584154] systemd[1]: Finished Coldplug All udev Devices. worker1 # [ 10.585327] systemd[1]: Reached target System Initialization. worker1 # [ 10.588256] systemd[1]: Started Discard unused filesystem blocks once a week. worker1 # [ 10.589921] systemd[1]: Started Daily Cleanup of Temporary Directories. worker2 # [ 10.604209] systemd[1]: Started Reset console on configuration changes. worker1 # [ 10.594827] systemd[1]: Reached target Timer Units. worker1 # [ 10.595652] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 10.606339] systemd[1]: Listening on Nix Daemon Socket. worker2 # [ 10.618785] systemd[1]: Starting resolvconf update... worker2 # [ 10.628557] systemd[1]: Starting D-Bus System Message Bus... worker1 # [ 10.618434] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 10.624218] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker1 # [ 10.635843] systemd[1]: Reached target Socket Units. worker1 # [ 10.640949] systemd[1]: Reached target Basic System. worker1 # [ 10.646512] systemd[1]: Starting Import lastlog data into lastlog2 database... worker1 # [ 10.655059] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 10.665805] systemd[1]: Starting Post-Boot Actions... worker2 # [ 10.681186] systemd[1]: Finished Post-Boot Actions. worker1 # [ 10.676507] systemd[1]: Started Reset console on configuration changes. worker1 # [ 10.685445] systemd[1]: Starting resolvconf update... worker1 # [ 10.694802] systemd[1]: Starting D-Bus System Message Bus... worker2 # [ 10.714721] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 10.722153] systemd[1]: Reached target Host and Network Name Lookups. client # [ 10.708491] (udev-worker)[354]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 10.729824] nsncd[481]: Sep 23 04:42:32.234 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 10.717270] (udev-worker)[352]: Network interface NamePolicy= disabled on kernel command line. client # [ 10.721749] (udev-worker)[354]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 10.745013] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 10.735393] systemd[1]: Finished Post-Boot Actions. worker2 # [ 10.751613] systemd[1]: Starting User Login Management... worker2 # [ 10.761856] dbus-broker-launch[485]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 10.771633] systemd[1]: Finished Import lastlog data into lastlog2 database. worker1 # [ 10.764432] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 10.776893] dbus-broker-launch[485]: NSS returned no entry for 'systemd-timesync' worker2 # [ 10.777937] dbus-broker-launch[485]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" lb # [ 10.738380] systemd[1]: Finished register-nix-paths.service. worker1 # [ 10.773247] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 10.795254] systemd[1]: Started D-Bus System Message Bus. worker1 # [ 10.783815] nsncd[482]: Sep 23 04:42:32.285 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 10.796275] systemd[1]: Reached target User and Group Name Lookups. lb # [ 10.766125] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. lb # [ 10.770051] systemd[1]: Finished Create SUID/SGID Wrappers. worker1 # [ 10.802437] systemd[1]: Starting User Login Management... lb # [ 10.773140] systemd[1]: Reached target System Initialization. lb # [ 10.774077] systemd[1]: Started Discard unused filesystem blocks once a week. lb # [ 10.775083] systemd[1]: Started niks3 garbage collection timer. lb # [ 10.775900] systemd[1]: Started Daily Cleanup of Temporary Directories. lb # [ 10.783835] systemd[1]: Reached target Timer Units. worker1 # [ 10.816930] systemd[1]: Finished Import lastlog data into lastlog2 database. lb # [ 10.786467] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 10.792534] systemd[1]: Listening on niks3 server socket. lb # [ 10.793406] systemd[1]: Listening on Nix Daemon Socket. lb # [ 10.794152] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. lb # [ 10.795316] systemd[1]: Reached target Socket Units. lb # [ 10.801939] systemd[1]: Reached target Basic System. worker2 # [ 10.847269] dbus-broker-launch[485]: Ready lb # [ 10.803069] systemd[1]: Starting Import lastlog data into lastlog2 database... worker1 # [ 10.837165] dbus-broker-launch[486]: Looking up NSS user entry for 'systemd-timesync'... lb # [ 10.810190] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 10.847187] dbus-broker-launch[486]: NSS returned no entry for 'systemd-timesync' lb # [ 10.816758] systemd[1]: Starting Post-Boot Actions... worker1 # [ 10.853491] dbus-broker-launch[486]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker1 # [ 10.870012] systemd[1]: Started D-Bus System Message Bus. lb # [ 10.843891] systemd[1]: Started Reset console on configuration changes. worker2 # [ 10.911305] systemd-logind[493]: New seat seat0. worker2 # [ 10.918710] systemd[1]: Started User Login Management. worker2 # [ 10.922987] systemd[1]: Starting linger-users.service... worker1 # [ 10.919057] dbus-broker-launch[486]: Ready worker2 # [ 10.937762] systemd[1]: Stopped target Host and Network Name Lookups. worker2 # [ 10.942708] systemd[1]: Stopping Host and Network Name Lookups... lb # [ 10.900277] systemd[1]: Starting resolvconf update... worker2 # [ 10.946713] systemd[1]: Stopped target User and Group Name Lookups. worker2 # [ 10.962065] systemd[1]: Stopping User and Group Name Lookups... worker2 # [ 10.970317] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker2 # [ 10.976244] systemd[1]: nscd.service: Deactivated successfully. lb # [ 10.935337] systemd[1]: Started rustfs.service. worker1 # [ 10.965963] systemd-logind[494]: New seat seat0. worker2 # [ 10.982866] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker1 # [ 10.975306] systemd[1]: Started User Login Management. worker2 # [ 10.987264] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 10.981394] systemd[1]: Starting linger-users.service... worker1 # [ 10.997381] systemd[1]: Stopped target Host and Network Name Lookups. lb # [ 10.964930] nsncd[442]: Sep 23 04:42:32.487 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 11.004471] systemd[1]: Stopping Host and Network Name Lookups... lb # [ 10.980152] systemd[1]: Starting rustfs-bucket.service... client # [ 11.006586] systemd[1]: Condition check resulted in Virtio network device being skipped. worker2 # [ 11.026968] systemd[1]: linger-users.service: Deactivated successfully. lb # [ 10.985955] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 11.031161] systemd[1]: Finished linger-users.service. client # [ 11.012144] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker1 # [ 11.023054] systemd[1]: Stopped target User and Group Name Lookups. client # [ 11.016119] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 11.017068] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. client # [ 11.018282] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker1 # [ 11.031257] systemd[1]: Stopping User and Group Name Lookups... client # [ 11.025235] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore lb # [ 11.002386] systemd[1]: Finished Post-Boot Actions. client # [ 11.026822] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 11.036167] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker1 # [ 11.039605] systemd[1]: nscd.service: Deactivated successfully. client # [ 11.036378] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 11.044728] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker1 # [ 11.050694] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 11.063257] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 11.067755] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 11.071937] nsncd[550]: Sep 23 04:42:32.583 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" lb # [ 11.031963] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. client # [ 11.075907] mousedev: PS/2 mouse device common for all mice worker2 # [ 11.080443] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 11.070532] systemd[1]: linger-users.service: Deactivated successfully. worker1 # [ 11.077883] systemd[1]: Finished linger-users.service. lb # [ 11.054728] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. client # [ 11.082110] systemd[1]: Finished register-nix-paths.service. worker2 # [ 11.108200] systemd[1]: Finished resolvconf update. lb # [ 11.064212] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 11.109138] systemd[1]: Reached target Preparation for Network. lb # [ 11.069879] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 11.117583] systemd[1]: Starting DHCP Client... worker1 # [ 11.108675] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 11.124401] systemd[1]: Starting Extra networking commands.... worker1 # [ 11.113174] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 11.117351] nsncd[551]: Sep 23 04:42:32.629 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 11.122912] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 11.140373] systemd[1]: Finished resolvconf update. worker1 # [ 11.148343] systemd[1]: Reached target Preparation for Network. lb # [ 11.119799] systemd[1]: Started backdoor.service. worker1 # [ 11.153513] systemd[1]: Starting DHCP Client... worker1 # [ 11.160195] systemd[1]: Starting Extra networking commands.... lb # [ 11.130913] systemd[1]: Starting D-Bus System Message Bus... lb # [ 11.186641] systemd[1]: Starting User Login Management... lb # [ 11.204938] systemd[1]: Finished Import lastlog data into lastlog2 database. worker2 # [ 11.338937] dhcpcd[582]: dhcpcd-10.3.2 starting worker2 # [ 11.349981] dhcpcd[620]: dev: loaded udev lb # [ 11.306104] (udev-worker)[346]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 11.355570] dhcpcd[620]: no valid interfaces found worker2 # [ 11.357970] dhcpcd[620]: no valid interfaces found worker2 # [ 11.360579] dhcpcd[620]: libudev: received NULL device worker2 # [ 11.363749] dhcpcd[620]: libudev: received NULL device lb # [ 11.331982] (udev-worker)[379]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 11.370427] dhcpcd[583]: dhcpcd-10.3.2 starting worker1 # [ 11.382364] dhcpcd[621]: dev: loaded udev worker1 # [ 11.386175] dhcpcd[621]: no valid interfaces found worker2 # [ 11.401922] systemd[1]: Finished Extra networking commands.. worker1 # [ 11.390215] dhcpcd[621]: no valid interfaces found worker1 # [ 11.392402] dhcpcd[621]: libudev: received NULL device worker2 # [ 11.405515] systemd[1]: Reached target Network. worker1 # [ 11.395166] dhcpcd[621]: libudev: received NULL device worker2 # [ 11.412182] systemd[1]: Starting Permit User Sessions... lb # [ 11.393616] (udev-worker)[379]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 11.427243] systemd[1]: Finished Extra networking commands.. worker1 # [ 11.429829] systemd[1]: Reached target Network. worker1 # [ 11.436313] systemd[1]: Starting Permit User Sessions... worker2 # [ 11.452231] systemd[1]: Finished Permit User Sessions. worker2 # [ 11.455336] systemd[1]: Started Getty on tty1. lb # connecting to host... worker2 # [ 11.459372] systemd[1]: Reached target Login Prompts. worker2 # [ 11.499806] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 worker1 # [ 11.478860] systemd[1]: Finished Permit User Sessions. worker1 # [ 11.481923] systemd[1]: Started Getty on tty1. worker1 # [ 11.484487] systemd[1]: Reached target Login Prompts. lb # [ 11.467677] systemd[1]: Stopped target Host and Network Name Lookups. worker1 # [ 11.512081] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 lb # [ 11.477177] systemd[1]: Stopping Host and Network Name Lookups... lb # [ 11.478349] systemd[1]: Stopped target User and Group Name Lookups. lb # [ 11.479209] systemd[1]: Stopping User and Group Name Lookups... client # [ 11.514129] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. client # [ 11.518147] systemd[1]: Finished Create SUID/SGID Wrappers. lb # [ 11.493805] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... lb # [ 11.494794] systemd[1]: nscd.service: Deactivated successfully. lb # [ 11.495969] dbus-broker-launch[472]: Looking up NSS user entry for 'systemd-timesync'... client # [ 11.522236] systemd[1]: Reached target System Initialization. client # [ 11.523179] systemd[1]: Started Discard unused filesystem blocks once a week. client # [ 11.526613] systemd[1]: Started Daily Cleanup of Temporary Directories. client # [ 11.527617] systemd[1]: Reached target Timer Units. client # [ 11.532774] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 11.533708] systemd[1]: Listening on Nix Daemon Socket. client # [ 11.534469] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. client # [ 11.535632] systemd[1]: Reached target Socket Units. lb # [ 11.511392] systemd[1]: Stopped Name Service Cache Daemon (nsncd). client # [ 11.545178] systemd[1]: Reached target Basic System. lb # [ 11.520330] dbus-broker-launch[472]: NSS returned no entry for 'systemd-timesync' client # [ 11.545978] systemd[1]: Started backdoor.service. client # [ 11.546668] systemd[1]: Starting Import lastlog data into lastlog2 database... lb # [ 11.521385] dbus-broker-launch[472]: Invalid user-name in /nix/store/4yiprls1sll47jid3xzq0m9nmi1qz2ww-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" client # [ 11.560652] systemd[1]: Starting nftables firewall... lb # [ 11.544811] systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 11.573938] systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 11.588102] systemd[1]: Starting Post-Boot Actions... lb # [ 11.561392] systemd-logind[474]: New seat seat0. lb # [ 11.581115] systemd[1]: Started User Login Management. lb # [ 11.581943] systemd[1]: Started D-Bus System Message Bus. client # [ 11.623316] systemd[1]: Started Reset console on configuration changes. lb # [ 11.615306] systemd[1]: Starting linger-users.service... client # [ 11.659374] systemd[1]: Starting resolvconf update... lb # [ 11.655121] dbus-broker-launch[472]: Ready client # connecting to host... lb # [ 11.721657] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 11.747872] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 11.751221] nsncd[454]: Sep 23 04:42:33.257 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" lb # [ 11.729011] nsncd[542]: Sep 23 04:42:33.252 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" lb # [ 11.739379] systemd[1]: Reached target Host and Network Name Lookups. client # [ 11.764435] systemd[1]: Reached target Host and Network Name Lookups. client # [ 11.765417] systemd[1]: Reached target User and Group Name Lookups. client: Guest shell says: b'Spawning backdoor root shell...\n' lb # [ 11.744330] systemd[1]: Reached target User and Group Name Lookups. lb # [ 11.749000] systemd[1]: linger-users.service: Deactivated successfully. worker2 # [ 11.804705] systemd[1]: Starting Virtual Console Setup... lb # [ 11.755629] systemd[1]: Finished linger-users.service. client # [ 11.804951] systemd[1]: Starting D-Bus System Message Bus... lb # [ 11.781232] systemd[1]: Finished resolvconf update. worker2 # [ 11.845493] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 11.830048] systemd[1]: Starting Virtual Console Setup... client # [ 11.826666] systemd[1]: Starting User Login Management... worker2 # [ 11.851407] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 11.852774] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 11.797572] systemd[1]: Starting DHCP Client... client # [ 11.834933] systemd[1]: Finished Post-Boot Actions. worker2 # [ 11.855457] systemd[1]: Starting Virtual Console Setup... worker1 # [ 11.856477] systemd[1]: Stopped Virtual Console Setup. client: connected to guest root shell client # [ 11.864399] systemd[1]: Finished Import lastlog data into lastlog2 database. worker1 # [ 11.860319] systemd[1]: Starting Virtual Console Setup... worker2 # [ 11.892163] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 11.889841] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client: (connecting took 12.31 seconds) client: (finished: waiting for the VM to finish booting, in 12.31 seconds) lb # [ 11.902619] systemd-logind[474]: Watching system buttons on /dev/input/event0 (gpio-keys) lb # [ 12.037804] systemd[1]: Condition check resulted in Virtio network device being skipped. client # [ 12.064610] dbus-broker-launch[480]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 12.098344] systemd-vconsole-setup[642]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 12.105993] systemd[1]: Finished Virtual Console Setup. worker1 # [ 12.111519] systemd-vconsole-setup[642]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 12.114467] systemd[1]: Finished Virtual Console Setup. client # [ 12.110106] systemd-logind[482]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) client # [ 12.116923] systemd-logind[482]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 12.118097] systemd-logind[482]: New seat seat0. lb # [ 12.099652] dhcpcd[593]: dhcpcd-10.3.2 starting client # [ 12.129070] systemd[1]: Started User Login Management. client # [ 12.135751] dbus-broker-launch[480]: NSS returned no entry for 'systemd-timesync' lb # [ 12.112995] dhcpcd[611]: dev: loaded udev client # [ 12.142340] dbus-broker-launch[480]: Invalid user-name in /nix/store/6wvgzhp3xj1v1xybf1ixpbqi5gif5fbv-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" client # [ 12.155497] systemd[1]: Stopped target Host and Network Name Lookups. client # [ 12.164203] systemd[1]: Stopping Host and Network Name Lookups... client # [ 12.165120] systemd[1]: Stopped target User and Group Name Lookups. client # [ 12.165999] systemd[1]: Stopping User and Group Name Lookups... client # [ 12.166799] systemd[1]: Starting linger-users.service... lb # [ 12.174393] mousedev: PS/2 mouse device common for all mice client # [ 12.183700] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker1 # [ 12.194672] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. client # [ 12.190366] systemd[1]: nscd.service: Deactivated successfully. client # [ 12.191278] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker2 # [ 12.210957] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker2 # [ 12.219836] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker1 # [ 12.207933] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. lb # [ 12.195214] 8021q: 802.1Q VLAN Support v1.8 worker2 # [ 12.224786] systemd[1]: Started backdoor.service. worker1 # [ 12.213258] systemd[1]: Started backdoor.service. client # [ 12.209697] systemd[1]: Started D-Bus System Message Bus. client # [ 12.251635] systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 12.261488] dbus-broker-launch[480]: Ready worker1 # connecting to host... worker2 # connecting to host... worker2 # [ 12.334911] 8021q: 802.1Q VLAN Support v1.8 worker1 # [ 12.331256] 8021q: 802.1Q VLAN Support v1.8 client # [ 12.318070] systemd[1]: linger-users.service: Deactivated successfully. client # [ 12.322580] systemd[1]: Finished linger-users.service. worker2 # [ 12.354632] (udev-worker)[646]: Network interface NamePolicy= disabled on kernel command line. client # [ 12.344244] systemd[1]: Finished resolvconf update. worker2 # [ 12.369075] (udev-worker)[648]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 12.371294] (udev-worker)[648]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 12.360503] (udev-worker)[655]: Network interface NamePolicy= disabled on kernel command line. client # [ 12.355930] systemd[1]: Starting DHCP Client... worker1 # [ 12.370310] (udev-worker)[651]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 12.375699] systemd[1]: Started Name Service Cache Daemon (nsncd). worker1 # [ 12.383097] (udev-worker)[651]: Network interface NamePolicy= disabled on kernel command line. lb # [ 12.371608] cfg80211: Loading compiled-in X.509 certificates for regulatory database client # [ 12.381076] nsncd[549]: Sep 23 04:42:33.886 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 12.383084] systemd[1]: Reached target Host and Network Name Lookups. client # [ 12.383973] systemd[1]: Reached target User and Group Name Lookups. lb # [ 12.380005] systemd-logind[474]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) worker2 # [ 12.452884] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker1 # [ 12.446853] cfg80211: Loading compiled-in X.509 certificates for regulatory database lb # [ 12.445975] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' lb # [ 12.446493] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' lb # [ 12.450243] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 lb # [ 12.450567] cfg80211: failed to load regulatory.db worker2 # [ 12.503862] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker2 # [ 12.504417] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker1 # [ 12.492175] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker1 # [ 12.492727] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker2 # [ 12.507348] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker2 # [ 12.507697] cfg80211: failed to load regulatory.db worker1 # [ 12.495710] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker1 # [ 12.496040] cfg80211: failed to load regulatory.db lb # [ 12.483141] systemd[1]: Finished Firewall. lb # [ 12.486750] systemd[1]: Reached target Preparation for Network. lb # [ 12.498949] systemd[1]: Starting Address configuration of eth1... lb # [ 12.511506] systemd[1]: Starting Extra networking commands.... client # [ 12.574156] dhcpcd[580]: dhcpcd-10.3.2 starting worker1 # [ 12.586662] systemd[1]: Condition check resulted in Virtio network device being skipped. client # [ 12.583736] dhcpcd[587]: dev: loaded udev worker1 # [ 12.591612] systemd[1]: Starting Address configuration of eth1... worker2 # [ 12.604390] systemd[1]: Condition check resulted in Virtio network device being skipped. worker2 # [ 12.612121] systemd[1]: Starting Address configuration of eth1... lb # [ 12.594523] 8021q: adding VLAN 0 to HW filter on device eth0 lb # [ 12.580752] dhcpcd[611]: eth0: waiting for carrier lb # [ 12.584667] dhcpcd[611]: libudev: received NULL device lb # [ 12.585475] dhcpcd[611]: libudev: received NULL device lb # [ 12.586590] dhcpcd[611]: eth0: carrier acquired client # [ 12.643912] 8021q: 802.1Q VLAN Support v1.8 lb # [ 12.607545] dhcpcd[611]: DUID 00:01:00:01:32:46:15:3a:52:54:00:12:34:56 lb # [ 12.612548] dhcpcd[611]: eth0: IAID 00:12:34:56 lb # [ 12.613269] dhcpcd[611]: eth0: adding address fe80::5054:ff:fe12:3456 worker2 # [ 12.683502] 8021q: adding VLAN 0 to HW filter on device eth0 worker2 # [ 12.673969] dhcpcd[620]: eth0: waiting for carrier worker2 # [ 12.674829] dhcpcd[620]: eth0: carrier acquired worker2 # [ 12.692757] dhcpcd[620]: DUID 00:01:00:01:32:46:15:3a:52:54:00:12:34:56 worker2 # [ 12.693804] dhcpcd[620]: eth0: IAID 00:12:34:56 worker2 # [ 12.694492] dhcpcd[620]: eth0: adding address fe80::5054:ff:fe12:3456 worker1 # [ 12.696728] 8021q: adding VLAN 0 to HW filter on device eth1 worker1 # [ 12.704149] 8021q: adding VLAN 0 to HW filter on device eth0 worker1 # [ 12.695877] dhcpcd[621]: eth0: waiting for carrier worker2 # [ 12.723906] 8021q: adding VLAN 0 to HW filter on device eth1 worker1 # [ 12.698871] dhcpcd[621]: eth0: carrier acquired worker1 # [ 12.699843] systemd-logind[494]: Watching system buttons on /dev/input/event0 (gpio-keys) worker1 # [ 12.710418] network-addresses-eth1-start[672]: adding address 192.168.1.3/24... done client # [ 12.718446] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker2 # [ 12.724515] network-addresses-eth1-start[674]: adding address 192.168.1.4/24... done worker1 # [ 12.715939] dhcpcd[621]: DUID 00:01:00:01:32:46:15:3a:52:54:00:12:34:56 worker1 # [ 12.719220] dhcpcd[621]: eth0: IAID 00:12:34:56 worker1 # [ 12.720775] dhcpcd[621]: eth0: adding address fe80::5054:ff:fe12:3456 worker2 # [ 12.737184] systemd-logind[493]: Watching system buttons on /dev/input/event0 (gpio-keys) worker1 # [ 12.726012] network-addresses-eth1-start[672]: adding address 2001:db8:1::3/64... done worker2 # [ 12.738404] network-addresses-eth1-start[674]: adding address 2001:db8:1::4/64... done lb # [ 12.717199] 8021q: adding VLAN 0 to HW filter on device eth1 worker2 # [ 12.755775] systemd[1]: Finished Address configuration of eth1. worker1 # [ 12.744466] systemd[1]: Finished Address configuration of eth1. client # [ 12.752609] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' client # [ 12.753122] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' client # [ 12.757986] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 client # [ 12.758294] cfg80211: failed to load regulatory.db lb # [ 12.746061] network-addresses-eth1-start[641]: adding address 192.168.1.2/24... done worker1 # [ 12.825344] mousedev: PS/2 mouse device common for all mice lb # [ 12.787049] network-addresses-eth1-start[641]: adding address 2001:db8:1::2/64... done client # [ 12.823712] systemd[1]: Finished nftables firewall. worker2 # [ 12.859855] mousedev: PS/2 mouse device common for all mice client # [ 12.827940] systemd[1]: Reached target Preparation for Network. client # [ 12.832803] systemd[1]: Starting Address configuration of eth1... client # [ 12.840090] systemd[1]: Starting Extra networking commands.... lb # [ 12.825482] systemd[1]: Finished Address configuration of eth1. client # [ 12.881387] 8021q: adding VLAN 0 to HW filter on device eth0 client # [ 12.867161] dhcpcd[587]: eth0: waiting for carrier client # [ 12.869387] dhcpcd[587]: libudev: received NULL device client # [ 12.870225] dhcpcd[587]: libudev: received NULL device client # [ 12.871126] dhcpcd[587]: eth0: carrier acquired client # [ 12.891981] dhcpcd[587]: DUID 00:01:00:01:32:46:15:3a:52:54:00:12:34:56 client # [ 12.893021] dhcpcd[587]: eth0: IAID 00:12:34:56 client # [ 12.893671] dhcpcd[587]: eth0: adding address fe80::5054:ff:fe12:3456 worker1 # [ 12.927707] systemd-logind[494]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) worker2 # [ 12.962910] systemd-logind[493]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) lb # [ 12.926622] systemd[1]: Finished Extra networking commands.. client # [ 12.969858] 8021q: adding VLAN 0 to HW filter on device eth1 lb # [ 12.932386] systemd[1]: Reached target Network. lb # [ 12.939076] systemd[1]: Starting PostgreSQL Server... lb # [ 12.952308] systemd[1]: Starting Permit User Sessions... client # [ 12.983294] network-addresses-eth1-start[621]: adding address 192.168.1.1/24... done client # [ 13.018788] network-addresses-eth1-start[621]: adding address 2001:db8:1::1/64... done client # [ 13.048436] systemd[1]: Finished Address configuration of eth1. lb # [ 13.045829] systemd[1]: Finished Permit User Sessions. lb # [ 13.058088] systemd[1]: Started Getty on tty1. lb # [ 13.061232] systemd[1]: Reached target Login Prompts. client # [ 13.122723] systemd[1]: Finished Extra networking commands.. client # [ 13.127443] systemd[1]: Reached target Network. client # [ 13.130785] systemd[1]: Starting Permit User Sessions... client # [ 13.227138] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 client # [ 13.214757] systemd[1]: Finished Permit User Sessions. client # [ 13.219431] systemd[1]: Started Getty on tty1. client # [ 13.220232] systemd[1]: Reached target Login Prompts. lb # [ 13.224997] postgresql-pre-start[723]: The files belonging to this database system will be owned by user "postgres". lb # [ 13.226485] postgresql-pre-start[723]: This user must also own the server process. lb # [ 13.237541] postgresql-pre-start[723]: The database cluster will be initialized with locale "en_US.UTF-8". lb # [ 13.240941] postgresql-pre-start[723]: The default database encoding has accordingly been set to "UTF8". lb # [ 13.242178] postgresql-pre-start[723]: The default text search configuration will be set to "english". lb # [ 13.243347] postgresql-pre-start[723]: Data page checksums are enabled. lb # [ 13.248221] postgresql-pre-start[723]: fixing permissions on existing directory /var/lib/postgresql/18 ... ok lb # [ 13.249572] postgresql-pre-start[723]: creating subdirectories ... ok lb # [ 13.250427] postgresql-pre-start[723]: selecting dynamic shared memory implementation ... posix lb # [ 13.365561] postgresql-pre-start[723]: selecting default "max_connections" ... 100 client: (finished: waiting for unit network-addresses-eth1.service, in 13.81 seconds) client: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; } worker1 # [ 13.471181] dhcpcd[621]: eth0: soliciting a DHCP lease worker1 # [ 13.480603] dhcpcd[621]: eth0: offered 10.0.2.15 from 10.0.2.2 worker1 # [ 13.488908] dhcpcd[621]: eth0: probing address 10.0.2.15/24 client # [ 13.490453] dhcpcd[587]: eth0: soliciting a DHCP lease client # [ 13.496702] dhcpcd[587]: eth0: offered 10.0.2.15 from 10.0.2.2 client: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; }, in 0.08 seconds) lb: waiting for unit network-addresses-eth1.service lb: waiting for the VM to finish booting lb: Guest shell says: b'Spawning backdoor root shell...\n' lb: connected to guest root shell lb: (connecting took 0.00 seconds) lb: (finished: waiting for the VM to finish booting, in 0.00 seconds) lb # [ 13.477454] postgresql-pre-start[723]: selecting default "shared_buffers" ... 128MB client # [ 13.504274] dhcpcd[587]: eth0: probing address 10.0.2.15/24 worker2 # [ 13.565496] dhcpcd[620]: eth0: soliciting a DHCP lease worker2 # [ 13.572657] dhcpcd[620]: eth0: offered 10.0.2.15 from 10.0.2.2 worker2 # [ 13.580297] dhcpcd[620]: eth0: probing address 10.0.2.15/24 lb: (finished: waiting for unit network-addresses-eth1.service, in 0.07 seconds) lb: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; } lb: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; }, in 0.05 seconds) worker1: waiting for unit network-addresses-eth1.service worker1: waiting for the VM to finish booting worker1: Guest shell says: b'Spawning backdoor root shell...\n' worker1: connected to guest root shell worker1: (connecting took 0.00 seconds) worker1: (finished: waiting for the VM to finish booting, in 0.00 seconds) client # [ 13.627041] systemd[1]: Starting Virtual Console Setup... client # [ 13.653123] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 13.657473] systemd[1]: Stopped Virtual Console Setup. client # [ 13.663728] systemd[1]: Starting Virtual Console Setup... client # [ 13.691903] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker1: (finished: waiting for unit network-addresses-eth1.service, in 0.08 seconds) worker1: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; } worker1: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; }, in 0.04 seconds) worker2: waiting for unit network-addresses-eth1.service worker2: waiting for the VM to finish booting worker2: Guest shell says: b'Spawning backdoor root shell...\n' worker2: connected to guest root shell worker2: (connecting took 0.00 seconds) worker2: (finished: waiting for the VM to finish booting, in 0.00 seconds) client # [ 13.742974] systemd-logind[482]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) client # [ 13.822454] systemd-vconsole-setup[699]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 13.826961] systemd[1]: Finished Virtual Console Setup. worker2: (finished: waiting for unit network-addresses-eth1.service, in 0.11 seconds) worker2: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; } worker2: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; }, in 0.04 seconds) lb: waiting for unit niks3.service worker1 # [ 14.041477] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 14.042602] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 14.043357] systemd[1]: Stopping Virtual Console Setup... worker1 # [ 14.055866] systemd[1]: Starting Virtual Console Setup... worker1 # [ 14.071528] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 14.075968] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 14.088734] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker1 # [ 14.079120] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker1 # [ 14.084150] systemd[1]: Starting Virtual Console Setup... worker2 # [ 14.095524] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 14.099114] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 14.099881] systemd[1]: Stopping Virtual Console Setup... worker2 # [ 14.108176] systemd[1]: Starting Virtual Console Setup... worker2 # [ 14.123829] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 14.128102] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 14.128874] systemd[1]: Starting Virtual Console Setup... worker1 # [ 14.142089] systemd-logind[494]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker1 # [ 14.147196] dhcpcd[621]: eth0: soliciting an IPv6 router worker1 # [ 14.147973] dhcpcd[621]: eth0: Router Advertisement from fe80::2 worker1 # [ 14.150337] dhcpcd[621]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker1 # [ 14.151452] dhcpcd[621]: eth0: adding route to fec0::/64 worker1 # [ 14.152510] dhcpcd[621]: eth0: adding default route via fe80::2 worker2 # [ 14.184475] systemd-logind[493]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker1 # [ 14.221632] systemd-vconsole-setup[705]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 14.226479] systemd[1]: Finished Virtual Console Setup. client # [ 14.230681] dhcpcd[587]: eth0: soliciting an IPv6 router client # [ 14.234279] dhcpcd[587]: eth0: Router Advertisement from fe80::2 client # [ 14.236753] dhcpcd[587]: eth0: adding address fec0::5054:ff:fe12:3456/64 client # [ 14.239261] dhcpcd[587]: eth0: adding route to fec0::/64 lb # [ 14.232626] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 client # [ 14.241349] dhcpcd[587]: eth0: adding default route via fe80::2 worker2 # [ 14.262480] systemd-vconsole-setup[705]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 14.265059] systemd[1]: Finished Virtual Console Setup. lb # [ 14.352428] dhcpcd[611]: eth0: soliciting a DHCP lease lb # [ 14.356532] dhcpcd[611]: eth0: offered 10.0.2.15 from 10.0.2.2 lb # [ 14.364273] dhcpcd[611]: eth0: probing address 10.0.2.15/24 worker2 # [ 14.443704] dhcpcd[620]: eth0: soliciting an IPv6 router worker2 # [ 14.447169] dhcpcd[620]: eth0: Router Advertisement from fe80::2 worker2 # [ 14.449950] dhcpcd[620]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker2 # [ 14.452532] dhcpcd[620]: eth0: adding route to fec0::/64 worker2 # [ 14.454571] dhcpcd[620]: eth0: adding default route via fe80::2 lb # [ 14.558884] systemd[1]: Starting Virtual Console Setup... lb # [ 14.580409] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 14.583642] systemd[1]: Stopped Virtual Console Setup. lb # [ 14.598544] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. lb # [ 14.605655] systemd[1]: Starting Virtual Console Setup... lb # [ 14.682684] systemd-logind[474]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) lb # [ 14.815197] systemd-vconsole-setup[753]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 14.818406] systemd[1]: Finished Virtual Console Setup. lb # [ 15.143429] postgresql-pre-start[723]: selecting default time zone ... UTC lb # [ 15.146571] postgresql-pre-start[723]: creating configuration files ... ok lb # [ 15.269370] dhcpcd[611]: eth0: soliciting an IPv6 router lb # [ 15.271003] dhcpcd[611]: eth0: Router Advertisement from fe80::2 lb # [ 15.271914] dhcpcd[611]: eth0: adding address fec0::5054:ff:fe12:3456/64 lb # [ 15.273486] dhcpcd[611]: eth0: adding route to fec0::/64 lb # [ 15.274237] dhcpcd[611]: eth0: adding default route via fe80::2 lb # [ 15.381306] postgresql-pre-start[723]: running bootstrap script ... ok lb # [ 15.933236] postgresql-pre-start[723]: performing post-bootstrap initialization ... ok lb # [ 16.088819] postgresql-pre-start[723]: syncing data to disk ... ok lb # [ 16.091042] postgresql-pre-start[723]: initdb: warning: enabling "trust" authentication for local connections lb # [ 16.092540] postgresql-pre-start[723]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. lb # [ 16.094662] postgresql-pre-start[723]: Success. You can now start the database server using: lb # [ 16.095714] postgresql-pre-start[723]: pg_ctl -D /var/lib/postgresql/18 -l logfile start lb # [ 16.206752] postgres[773]: [773] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit lb # [ 16.211924] postgres[773]: [773] LOG: listening on IPv6 address "::1", port 5432 lb # [ 16.214497] postgres[773]: [773] LOG: listening on IPv4 address "127.0.0.1", port 5432 lb # [ 16.217491] postgres[773]: [773] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432" lb # [ 16.234523] postgres[790]: [790] LOG: database system was shut down at 2026-09-23 04:42:37 GMT lb # [ 16.241731] postgres[773]: [773] LOG: database system is ready to accept connections lb # [ 16.247456] systemd[1]: Started PostgreSQL Server. lb # [ 16.256481] systemd[1]: Starting PostgreSQL Setup Scripts... lb # [ 16.434694] postgresql-setup-start[802]: CREATE DATABASE lb # [ 16.474243] postgresql-setup-start[807]: CREATE ROLE lb # [ 16.491681] postgresql-setup-start[809]: ALTER DATABASE lb # [ 16.497562] systemd[1]: Finished PostgreSQL Setup Scripts. lb # [ 16.499522] systemd[1]: Reached target PostgreSQL. worker2 # [ 18.050555] dhcpcd[620]: eth0: leased 10.0.2.15 for 86400 seconds worker2 # [ 18.053888] dhcpcd[620]: eth0: adding route to 10.0.2.0/24 worker2 # [ 18.060297] dhcpcd[620]: eth0: adding default route via 10.0.2.2 worker2 # [ 18.188902] systemd[1]: Started DHCP Client. worker2 # [ 18.190931] systemd[1]: Reached target Multi-User System. worker2 # [ 18.191739] systemd[1]: Startup finished in 1.050s (kernel) + 4.372s (initrd) + 12.767s (userspace) = 18.190s. worker1 # [ 18.319368] dhcpcd[621]: eth0: leased 10.0.2.15 for 86400 seconds worker1 # [ 18.323547] dhcpcd[621]: eth0: adding route to 10.0.2.0/24 worker1 # [ 18.330930] dhcpcd[621]: eth0: adding default route via 10.0.2.2 worker1 # [ 18.473125] systemd[1]: Started DHCP Client. worker1 # [ 18.474326] systemd[1]: Reached target Multi-User System. worker1 # [ 18.475148] systemd[1]: Startup finished in 1.049s (kernel) + 4.370s (initrd) + 13.054s (userspace) = 18.474s. client # [ 18.558532] dhcpcd[587]: eth0: leased 10.0.2.15 for 86400 seconds client # [ 18.561788] dhcpcd[587]: eth0: adding route to 10.0.2.0/24 client # [ 18.566331] dhcpcd[587]: eth0: adding default route via 10.0.2.2 client # [ 18.705775] systemd[1]: Started DHCP Client. client # [ 18.707860] systemd[1]: Reached target Multi-User System. client # [ 18.709250] systemd[1]: Startup finished in 1.060s (kernel) + 5.097s (initrd) + 12.551s (userspace) = 18.708s. lb # [ 19.245325] dhcpcd[611]: eth0: leased 10.0.2.15 for 86400 seconds lb # [ 19.247371] dhcpcd[611]: eth0: adding route to 10.0.2.0/24 lb # [ 19.249200] dhcpcd[611]: eth0: adding default route via 10.0.2.2 lb # [ 19.360331] systemd[1]: Started DHCP Client. lb # [ 19.363432] systemd[1]: Reached target Network is Online. lb # [ 19.368610] systemd[1]: Started Envoy reverse proxy. lb # [ 19.380656] systemd[1]: Started mock-oidc.service. lb # [ 19.889855] mock-oidc-server[892]: Mock OIDC Server running lb # [ 19.892415] mock-oidc-server[892]: OIDC Address: 192.168.1.2:8080 lb # [ 19.893687] mock-oidc-server[892]: Issue Address: 0.0.0.0:8081 lb # [ 19.894808] mock-oidc-server[892]: Issuer: http://192.168.1.2:8080/oidc lb # [ 19.896058] mock-oidc-server[892]: JWKS: http://192.168.1.2:8080/oidc/.well-known/jwks.json lb # [ 19.897479] mock-oidc-server[892]: Discovery: http://192.168.1.2:8080/oidc/.well-known/openid-configuration lb # [ 19.899095] mock-oidc-server[892]: Issue tokens: http://0.0.0.0:8081/issue?sub=... lb # [ 19.941424] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:467] initializing epoch 0 (base id=0, hot restart version=11.120) lb # [ 19.943340] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:469] statically linked extensions: lb # [ 19.945061] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.health_check.event_sinks: envoy.health_check.event_sink.file lb # [ 19.947174] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator lb # [ 19.950058] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest lb # [ 19.956242] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.filters.listener: envoy.filters.listener.dynamic_modules, envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.set_filter_state, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector lb # [ 19.985485] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.matching.action: envoy.matching.action.transport_socket.name, envoy.matching.actions.format_string, filter-chain-name lb # [ 19.988185] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 19.990280] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_mappers: envoy.tls.certificate_mappers.static_name, envoy.tls.upstream_certificate_mappers.filter_state_override lb # [ 19.993325] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.http.forward_client_cert_details: forward_client_cert lb # [ 19.995566] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor lb # [ 19.998716] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.filters.upstream_network: envoy.filters.upstream_network.reverse_tunnel_lifecycle lb # [ 20.001742] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.quic.connection_id_generator: envoy.quic.connection_id_generator.quic_lb, envoy.quic.deterministic_connection_id_generator lb # [ 20.005568] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.http.injected_credentials: envoy.http.injected_credentials.generic, envoy.http.injected_credentials.oauth2 lb # [ 20.007926] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.rbac.principals: envoy.rbac.principals.mtls_authenticated lb # [ 20.010138] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.ext_authz, envoy.filters.udp.session.http_capsule lb # [ 20.013755] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.geoip_providers: envoy.geoip_providers.maxmind lb # [ 20.015843] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.credential_injector, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec, envoy.filters.http.upstream_rbac, envoy.filters.http.wasm lb # [ 20.023850] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.generic_proxy.filters: envoy.filters.generic.router lb # [ 20.026246] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.upstream_options: envoy.extensions.upstreams.http.reverse_tunnel.v3.ReverseTunnelUpstreamCodecOptions, envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options lb # [ 20.031204] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector, envoy.upstream.local_address_selector.filter_state_override lb # [ 20.034818] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.matching.stat_matching_data.input: stat_full_name_match_input lb # [ 20.036861] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.bootstrap: envoy.bootstrap.dynamic_modules, envoy.bootstrap.internal_listener, envoy.bootstrap.reverse_tunnel.downstream_socket_interface, envoy.bootstrap.reverse_tunnel.upstream_socket_interface, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface, envoy.extensions.network.socket_interface.sockmap lb # [ 20.042341] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.matching.transport_socket.input: envoy.matching.inputs.endpoint_metadata, envoy.matching.inputs.locality_metadata, envoy.matching.inputs.transport_socket_filter_state lb # [ 20.046603] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso lb # [ 20.050536] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher lb # [ 20.052611] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.resource_monitors: envoy.resource_monitors.cgroup_memory, envoy.resource_monitors.cpu_utilization, envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource lb # [ 20.055702] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter lb # [ 20.058228] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.rds_factory: envoy.rds_factory.default lb # [ 20.060133] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter lb # [ 20.062372] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.regex_engines: envoy.regex_engines.google_re2 lb # [ 20.064168] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts lb # [ 20.067086] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme lb # [ 20.070783] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.matching.input_matchers: envoy.matching.input_matchers.generic_request_matcher, envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.dynamic_modules, envoy.matching.matchers.ip, envoy.matching.matchers.metadata_matcher, envoy.matching.matchers.runtime_fraction lb # [ 20.075721] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.health_checkers: envoy.health_checkers.dynamic_modules, envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift lb # [ 20.079318] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation lb # [ 20.081513] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.retry_priorities: envoy.retry_priorities.previous_priorities lb # [ 20.083346] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.access_loggers: envoy.access_loggers.dynamic_modules, envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stats, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log lb # [ 20.090955] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable lb # [ 20.093102] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.quic.client_packet_writer: envoy.quic.packet_writer.default lb # [ 20.095037] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.srds_factory: envoy.srds_factory.default lb # [ 20.096836] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel, envoy.access_loggers.extension_filters.process_ratelimit lb # [ 20.099517] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.fault, envoy.filters.http.a2a, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.ai_protocol_manager, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.api_key_auth, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.bandwidth_share, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cache_v2, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_server, envoy.filters.http.file_system_buffer, envoy.filters.http.filter_chain, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_reverse_transcoder, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.mcp, envoy.filters.http.mcp_json_rest_bridge, envoy.filters.http.mcp_router, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.proto_api_scrubber, envoy.filters.http.proto_message_extraction, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.sse_to_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.thrift_to_metadata, envoy.filters.http.transform, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router lb # [ 20.131358] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.hashable_string, envoy.network.application_protocols, envoy.network.ip, envoy.network.network_namespace, envoy.network.transport_socket.http_11_proxy.info, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.ratelimit.hits_addend, envoy.router.debug_config, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port lb # [ 20.140269] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr lb # [ 20.142369] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy lb # [ 20.145354] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche lb # [ 20.147553] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.envelope, envoy.http.stateful_session.header lb # [ 20.150514] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.quic.server_preferred_address: quic.server_preferred_address.datasource, quic.server_preferred_address.fixed lb # [ 20.153807] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.http.cache_v2: envoy.extensions.http.cache_v2.file_system_http_cache, envoy.extensions.http.cache_v2.simple lb # [ 20.157986] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory lb # [ 20.161375] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.route_config_update_requester: envoy.route_config_update_requester.default lb # [ 20.163637] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] network.connection.client: default, envoy_internal lb # [ 20.166024] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.protocols: dubbo lb # [ 20.170058] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.upstreams: envoy.filters.connection_pools.tcp.generic lb # [ 20.171892] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.built_in_formatters: envoy.built_in_formatters.cel, envoy.built_in_formatters.http.default, envoy.built_in_formatters.metadata, envoy.built_in_formatters.stream_info.default, envoy.built_in_formatters.xfcc_value worker2 # [ 20.218757] systemd[1]: Reached target Network is Online. lb # [ 20.176169] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo, envoy.network.dns_resolver.hickory worker1 # [ 20.213200] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 20.178732] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls worker2 # [ 20.230137] systemd[1]: Starting scheduler-token.service... lb # [ 20.186344] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.string_matcher: envoy.string_matcher.lua lb # [ 20.189743] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 20.192754] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.dynamic_metadata, envoy.matching.inputs.dynamic_module_data_input, envoy.matching.inputs.filter_state, envoy.matching.inputs.local_reply, envoy.matching.inputs.network_namespace, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params lb # [ 20.203696] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.matching.stat_tag_matching_data.input: stat_tag_value_input lb # [ 20.205670] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.grpc_credentials: envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata lb # [ 20.207862] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.thrift_proxy.transports: auto, framed, header, unframed lb # [ 20.209785] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.tracers: envoy.tracers.datadog, envoy.tracers.dynamic_modules, envoy.tracers.fluentd, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin lb # [ 20.213113] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.quic.connection_debug_visitor: envoy.quic.connection_debug_visitor.basic, envoy.quic.connection_debug_visitor.quic_stats lb # [ 20.215546] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.clusters: envoy.cluster.dns, envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.composite, envoy.clusters.dynamic_forward_proxy, envoy.clusters.dynamic_modules, envoy.clusters.mcp_multicluster, envoy.clusters.redis, envoy.clusters.reverse_connection lb # [ 20.220784] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.matching.generic_proxy_request_input.input: envoy.matching.generic_proxy.input.host, envoy.matching.generic_proxy.input.method, envoy.matching.generic_proxy.input.path, envoy.matching.generic_proxy.input.property, envoy.matching.generic_proxy.input.request, envoy.matching.generic_proxy.input.service lb # [ 20.225170] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.load_balancing_policies: envoy.load_balancing_policies.client_side_weighted_round_robin, envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.dynamic_modules, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.load_aware_locality, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.override_host, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset, envoy.load_balancing_policies.wrr_locality lb # [ 20.232826] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.common.key_value: envoy.key_value.file_based lb # [ 20.234551] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.request_id: envoy.request_id.uuid lb # [ 20.236171] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.generic_proxy.codecs: envoy.generic_proxy.codecs.dubbo, envoy.generic_proxy.codecs.http1 lb # [ 20.238355] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.http.ext_proc.response_processors: envoy.extensions.http.ext_proc.save_processing_response lb # [ 20.240553] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.route.early_data_policy: envoy.route.early_data_policy.default lb # [ 20.242417] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8 lb # [ 20.244376] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain lb # [ 20.247475] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.dynamic_modules, envoy.tls.cert_validator.spiffe lb # [ 20.250307] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router lb # [ 20.253409] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router lb # [ 20.255129] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor lb # [ 20.257817] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.content_parsers: envoy.content_parsers.json lb # [ 20.259503] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.dynamic_modules, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd lb # [ 20.264910] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.dynamic_modules, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.ext_proc, envoy.filters.network.generic_proxy, envoy.filters.network.geoip, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.reverse_tunnel, envoy.filters.network.reverse_tunnel_drain_aware_http_connection_manager, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_bandwidth_limit, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy lb # [ 20.278816] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port lb # [ 20.281449] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.serializers: dubbo.hessian2 lb # [ 20.285507] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation lb # [ 20.288089] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple lb # [ 20.290618] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.http.header_validators: envoy.http.header_validators.envoy_default lb # [ 20.292568] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.ssl.server_context_factory: envoy.ssl.server_context_factory.default lb # [ 20.294451] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher worker2 # [ 20.341183] systemd[1]: scheduler-token.service: Deactivated successfully. lb # [ 20.297033] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.network_namespace, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san worker2 # [ 20.347169] systemd[1]: Finished scheduler-token.service. lb # [ 20.304043] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.tls.certificate_mappers: envoy.tls.certificate_mappers.sni, envoy.tls.certificate_mappers.static_name lb # [ 20.306307] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] envoy.connection_handler: envoy.connection_handler.default lb # [ 20.308065] envoy[891]: [2026-09-23 04:42:41.470][891][info][main] [source/server/server.cc:471] quic.http_server_connection: quic.http_server_connection.default lb # [ 20.309849] envoy[891]: [2026-09-23 04:42:41.471][891][info][main] [source/server/server.cc:471] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action worker2 # [ 20.353228] systemd[1]: scheduler-token.service: Consumed 66ms CPU time over 115ms wall clock time, 3.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb # [ 20.311897] envoy[891]: [2026-09-23 04:42:41.471][891][info][main] [source/server/server.cc:471] envoy.resolvers: envoy.ip, envoy.resolvers.reverse_connection lb # [ 20.313817] envoy[891]: [2026-09-23 04:42:41.471][891][info][main] [source/server/server.cc:471] envoy.http.ext_proc.processing_request_modifiers: envoy.extensions.http.ext_proc.mapped_attribute_builder worker2 # [ 20.362002] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 20.316019] envoy[891]: [2026-09-23 04:42:41.471][891][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment, envoy.tracers.opentelemetry.resource_detectors.static_config lb # [ 20.319486] envoy[891]: [2026-09-23 04:42:41.471][891][info][main] [source/server/server.cc:471] envoy.tls.certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 20.321585] envoy[891]: [2026-09-23 04:42:41.471][891][info][main] [source/server/server.cc:471] envoy.formatter: envoy.formatter.cel, envoy.formatter.dynamic_modules, envoy.formatter.file_content, envoy.formatter.generic_secret, envoy.formatter.metadata, envoy.formatter.req_without_query lb # [ 20.324903] envoy[891]: [2026-09-23 04:42:41.471][891][info][main] [source/server/server.cc:471] envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua, envoy.router.cluster_specifier_plugin.matcher lb # [ 20.327396] envoy[891]: [2026-09-23 04:42:41.471][891][info][main] [source/server/server.cc:471] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.dynamic_modules, envoy.filters.udp_listener.udp_proxy lb # [ 20.330134] envoy[891]: [2026-09-23 04:42:41.471][891][info][main] [source/server/server.cc:471] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff lb # [ 20.332574] envoy[891]: [2026-09-23 04:42:41.471][891][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.cel, envoy.tracers.opentelemetry.samplers.dynatrace, envoy.tracers.opentelemetry.samplers.parent_based, envoy.tracers.opentelemetry.samplers.trace_id_ratio_based lb # [ 20.336797] envoy[891]: [2026-09-23 04:42:41.471][891][info][main] [source/server/server.cc:471] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case lb # [ 20.339061] envoy[891]: [2026-09-23 04:42:41.471][891][info][main] [source/server/server.cc:471] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls lb # [ 20.343990] envoy[891]: [2026-09-23 04:42:41.496][891][info][main] [source/server/server.cc:531] HTTP header map info: lb # [ 20.345354] envoy[891]: [2026-09-23 04:42:41.496][891][info][main] [source/server/server.cc:534] request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-host,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-request-id lb # [ 20.361107] envoy[891]: [2026-09-23 04:42:41.497][891][info][main] [source/server/server.cc:534] request trailer map: 120 bytes: worker1 # [ 20.399983] nix-grpc-daemon[770]: ts=2026-09-23T04:42:41Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" lb # [ 20.362586] envoy[891]: [2026-09-23 04:42:41.497][891][info][main] [source/server/server.cc:534] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id lb # [ 20.371647] envoy[891]: [2026-09-23 04:42:41.497][891][info][main] [source/server/server.cc:534] response trailer map: 144 bytes: grpc-message,grpc-status lb # [ 20.373512] envoy[891]: [2026-09-23 04:42:41.531][891][info][main] [source/server/server.cc:968] runtime: {} lb # [ 20.374779] envoy[891]: [2026-09-23 04:42:41.535][891][info][main] [source/server/server.cc:420] Raised file descriptor limits from 1024 to 524288. lb # [ 20.376458] envoy[891]: [2026-09-23 04:42:41.535][891][info][main] [source/server/server.cc:796] Starting admin HTTP server at 127.0.0.1:9901 worker1 # [ 20.409511] nix-grpc-daemon[770]: ts=2026-09-23T04:42:41Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc lb # [ 20.378049] envoy[891]: [2026-09-23 04:42:41.535][891][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:9901 lb # [ 20.379544] envoy[891]: [2026-09-23 04:42:41.540][891][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration worker1 # [ 20.411358] nix-grpc-daemon[770]: ts=2026-09-23T04:42:41Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 lb # [ 20.381250] envoy[891]: [2026-09-23 04:42:41.540][891][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s) lb # [ 20.382812] envoy[891]: [2026-09-23 04:42:41.540][891][info][config] [source/server/configuration_impl.cc:138] loading 2 cluster(s) lb # [ 20.384358] envoy[891]: [2026-09-23 04:42:41.699][891][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s) lb # [ 20.385879] envoy[891]: [2026-09-23 04:42:41.718][891][info][config] [source/server/configuration_impl.cc:164] loading stats configuration worker1 # [ 20.420767] nix-grpc-daemon[770]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR lb # [ 20.387471] envoy[891]: [2026-09-23 04:42:41.719][891][warning][main] [source/server/server.cc:1038] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor. lb # [ 20.390838] envoy[891]: [2026-09-23 04:42:41.719][891][info][main] [source/server/server.cc:1077] starting main dispatch loop worker1 # [ 20.423432] nix-grpc-daemon[770]: E0000 00:00:1790138561.940062 770 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 20.427926] nix-grpc-daemon[770]: E0000 00:00:1790138561.947459 770 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb # [ 20.421974] envoy[891]: [2026-09-23 04:42:41.951][891][info][runtime] [source/common/runtime/runtime_impl.cc:567] RTDS has finished initialization lb # [ 20.423811] envoy[891]: [2026-09-23 04:42:41.951][891][info][upstream] [source/common/upstream/cluster_manager_impl.cc:250] cm init: all clusters initialized lb # [ 20.425840] envoy[891]: [2026-09-23 04:42:41.951][891][info][main] [source/server/server.cc:1057] all clusters initialized. initializing init manager lb # [ 20.430299] envoy[891]: [2026-09-23 04:42:41.959][891][info][config] [source/common/listener_manager/listener_manager_impl.cc:1067] all dependencies initialized. starting workers worker1 # [ 20.447224] nix-grpc-daemon[770]: ts=2026-09-23T04:42:41Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 20.472106] nix-grpc-daemon[770]: E0000 00:00:1790138561.964193 813 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 20.474676] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 20.475518] nix-grpc-daemon[770]: E0000 00:00:1790138561.985792 813 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 20.489963] nix-grpc-daemon[770]: E0000 00:00:1790138561.986365 813 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 20.493100] nix-grpc-daemon[770]: E0000 00:00:1790138562.008381 813 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 20.501726] nix-grpc-daemon[770]: ts=2026-09-23T04:42:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.503814] nix-grpc-daemon[770]: ts=2026-09-23T04:42:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.521326] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 20.529253] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 20.531488] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 20.541388] nix-grpc-daemon[773]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 20.544134] nix-grpc-daemon[773]: E0000 00:00:1790138562.059788 773 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 20.548913] nix-grpc-daemon[773]: E0000 00:00:1790138562.067300 773 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 20.565222] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 20.567852] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 20.574041] nix-grpc-daemon[773]: E0000 00:00:1790138562.091331 815 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 20.604123] nix-grpc-daemon[773]: E0000 00:00:1790138562.110505 815 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 20.611681] nix-grpc-daemon[773]: E0000 00:00:1790138562.111053 815 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 20.615112] nix-grpc-daemon[773]: E0000 00:00:1790138562.129394 815 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 20.626932] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.628827] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.705306] nix-grpc-daemon[770]: ts=2026-09-23T04:42:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.710692] nix-grpc-daemon[770]: ts=2026-09-23T04:42:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.830046] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.832815] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.911797] nix-grpc-daemon[770]: ts=2026-09-23T04:42:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.917325] nix-grpc-daemon[770]: ts=2026-09-23T04:42:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.033578] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.035754] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.118289] nix-grpc-daemon[770]: ts=2026-09-23T04:42:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.123914] nix-grpc-daemon[770]: ts=2026-09-23T04:42:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.237352] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.242725] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.324941] nix-grpc-daemon[770]: ts=2026-09-23T04:42:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.330266] nix-grpc-daemon[770]: ts=2026-09-23T04:42:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.444529] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.449876] nix-grpc-daemon[773]: ts=2026-09-23T04:42:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.531323] nix-grpc-daemon[770]: ts=2026-09-23T04:42:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.533654] nix-grpc-daemon[770]: ts=2026-09-23T04:42:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.651159] nix-grpc-daemon[773]: ts=2026-09-23T04:42:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.653721] nix-grpc-daemon[773]: ts=2026-09-23T04:42:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.734710] nix-grpc-daemon[770]: ts=2026-09-23T04:42:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.740281] nix-grpc-daemon[770]: ts=2026-09-23T04:42:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.855162] nix-grpc-daemon[773]: ts=2026-09-23T04:42:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.860835] nix-grpc-daemon[773]: ts=2026-09-23T04:42:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.941505] nix-grpc-daemon[770]: ts=2026-09-23T04:42:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.947002] nix-grpc-daemon[770]: ts=2026-09-23T04:42:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.062148] nix-grpc-daemon[773]: ts=2026-09-23T04:42:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.068003] nix-grpc-daemon[773]: ts=2026-09-23T04:42:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.148160] nix-grpc-daemon[770]: ts=2026-09-23T04:42:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.153546] nix-grpc-daemon[770]: ts=2026-09-23T04:42:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.268776] nix-grpc-daemon[773]: ts=2026-09-23T04:42:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.270997] nix-grpc-daemon[773]: ts=2026-09-23T04:42:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.354277] nix-grpc-daemon[770]: ts=2026-09-23T04:42:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.357688] nix-grpc-daemon[770]: ts=2026-09-23T04:42:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.471949] nix-grpc-daemon[773]: ts=2026-09-23T04:42:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.474140] nix-grpc-daemon[773]: ts=2026-09-23T04:42:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.560626] nix-grpc-daemon[770]: ts=2026-09-23T04:42:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.564419] nix-grpc-daemon[770]: ts=2026-09-23T04:42:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.675753] nix-grpc-daemon[773]: ts=2026-09-23T04:42:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.678790] nix-grpc-daemon[773]: ts=2026-09-23T04:42:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.761386] nix-grpc-daemon[770]: ts=2026-09-23T04:42:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.766305] nix-grpc-daemon[770]: ts=2026-09-23T04:42:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.879868] nix-grpc-daemon[773]: ts=2026-09-23T04:42:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.883074] nix-grpc-daemon[773]: ts=2026-09-23T04:42:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.967077] nix-grpc-daemon[770]: ts=2026-09-23T04:42:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.971135] nix-grpc-daemon[770]: ts=2026-09-23T04:42:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.084147] nix-grpc-daemon[773]: ts=2026-09-23T04:42:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.086963] nix-grpc-daemon[773]: ts=2026-09-23T04:42:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.172175] nix-grpc-daemon[770]: ts=2026-09-23T04:42:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.177322] nix-grpc-daemon[770]: ts=2026-09-23T04:42:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.288653] nix-grpc-daemon[773]: ts=2026-09-23T04:42:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.294215] nix-grpc-daemon[773]: ts=2026-09-23T04:42:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.378283] nix-grpc-daemon[770]: ts=2026-09-23T04:42:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.383095] nix-grpc-daemon[770]: ts=2026-09-23T04:42:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.495707] nix-grpc-daemon[773]: ts=2026-09-23T04:42:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.501640] nix-grpc-daemon[773]: ts=2026-09-23T04:42:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.586116] nix-grpc-daemon[770]: ts=2026-09-23T04:42:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.590576] nix-grpc-daemon[770]: ts=2026-09-23T04:42:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.704879] nix-grpc-daemon[773]: ts=2026-09-23T04:42:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.709577] nix-grpc-daemon[773]: ts=2026-09-23T04:42:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.786922] nix-grpc-daemon[770]: ts=2026-09-23T04:42:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.792716] nix-grpc-daemon[770]: ts=2026-09-23T04:42:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.911161] nix-grpc-daemon[773]: ts=2026-09-23T04:42:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.916881] nix-grpc-daemon[773]: ts=2026-09-23T04:42:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.993661] nix-grpc-daemon[770]: ts=2026-09-23T04:42:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.998656] nix-grpc-daemon[770]: ts=2026-09-23T04:42:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.118036] nix-grpc-daemon[773]: ts=2026-09-23T04:42:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.121415] nix-grpc-daemon[773]: ts=2026-09-23T04:42:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.200326] nix-grpc-daemon[770]: ts=2026-09-23T04:42:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.206787] nix-grpc-daemon[770]: ts=2026-09-23T04:42:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.323401] nix-grpc-daemon[773]: ts=2026-09-23T04:42:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.329761] nix-grpc-daemon[773]: ts=2026-09-23T04:42:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.407982] nix-grpc-daemon[770]: ts=2026-09-23T04:42:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.414572] nix-grpc-daemon[770]: ts=2026-09-23T04:42:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.531128] nix-grpc-daemon[773]: ts=2026-09-23T04:42:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.536815] nix-grpc-daemon[773]: ts=2026-09-23T04:42:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.617513] nix-grpc-daemon[770]: ts=2026-09-23T04:42:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.621843] nix-grpc-daemon[770]: ts=2026-09-23T04:42:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.740002] nix-grpc-daemon[773]: ts=2026-09-23T04:42:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.744365] nix-grpc-daemon[773]: ts=2026-09-23T04:42:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.818129] nix-grpc-daemon[770]: ts=2026-09-23T04:42:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.823786] nix-grpc-daemon[770]: ts=2026-09-23T04:42:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.941509] nix-grpc-daemon[773]: ts=2026-09-23T04:42:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.947191] nix-grpc-daemon[773]: ts=2026-09-23T04:42:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.024859] nix-grpc-daemon[770]: ts=2026-09-23T04:42:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.030476] nix-grpc-daemon[770]: ts=2026-09-23T04:42:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.148773] nix-grpc-daemon[773]: ts=2026-09-23T04:42:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.154296] nix-grpc-daemon[773]: ts=2026-09-23T04:42:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.231387] nix-grpc-daemon[770]: ts=2026-09-23T04:42:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.237013] nix-grpc-daemon[770]: ts=2026-09-23T04:42:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.355924] nix-grpc-daemon[773]: ts=2026-09-23T04:42:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.361634] nix-grpc-daemon[773]: ts=2026-09-23T04:42:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.437937] nix-grpc-daemon[770]: ts=2026-09-23T04:42:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.443725] nix-grpc-daemon[770]: ts=2026-09-23T04:42:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.563118] nix-grpc-daemon[773]: ts=2026-09-23T04:42:47Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.568968] nix-grpc-daemon[773]: ts=2026-09-23T04:42:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.646868] nix-grpc-daemon[770]: ts=2026-09-23T04:42:47Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.651376] nix-grpc-daemon[770]: ts=2026-09-23T04:42:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.772765] nix-grpc-daemon[773]: ts=2026-09-23T04:42:47Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.777118] nix-grpc-daemon[773]: ts=2026-09-23T04:42:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.848508] nix-grpc-daemon[770]: ts=2026-09-23T04:42:47Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.858823] nix-grpc-daemon[770]: ts=2026-09-23T04:42:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.973984] nix-grpc-daemon[773]: ts=2026-09-23T04:42:47Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.979870] nix-grpc-daemon[773]: ts=2026-09-23T04:42:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.054627] nix-grpc-daemon[770]: ts=2026-09-23T04:42:47Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.060564] nix-grpc-daemon[770]: ts=2026-09-23T04:42:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.181431] nix-grpc-daemon[773]: ts=2026-09-23T04:42:47Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.187108] nix-grpc-daemon[773]: ts=2026-09-23T04:42:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.261562] nix-grpc-daemon[770]: ts=2026-09-23T04:42:47Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.267268] nix-grpc-daemon[770]: ts=2026-09-23T04:42:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.388676] nix-grpc-daemon[773]: ts=2026-09-23T04:42:47Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.394307] nix-grpc-daemon[773]: ts=2026-09-23T04:42:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.468560] nix-grpc-daemon[770]: ts=2026-09-23T04:42:47Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.474200] nix-grpc-daemon[770]: ts=2026-09-23T04:42:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.595522] nix-grpc-daemon[773]: ts=2026-09-23T04:42:48Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.601106] nix-grpc-daemon[773]: ts=2026-09-23T04:42:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.676746] nix-grpc-daemon[770]: ts=2026-09-23T04:42:48Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.681749] nix-grpc-daemon[770]: ts=2026-09-23T04:42:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.804746] nix-grpc-daemon[773]: ts=2026-09-23T04:42:48Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.808996] nix-grpc-daemon[773]: ts=2026-09-23T04:42:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.882882] nix-grpc-daemon[770]: ts=2026-09-23T04:42:48Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.888666] nix-grpc-daemon[770]: ts=2026-09-23T04:42:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.005911] nix-grpc-daemon[773]: ts=2026-09-23T04:42:48Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.011979] nix-grpc-daemon[773]: ts=2026-09-23T04:42:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.089556] nix-grpc-daemon[770]: ts=2026-09-23T04:42:48Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.095152] nix-grpc-daemon[770]: ts=2026-09-23T04:42:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.213488] nix-grpc-daemon[773]: ts=2026-09-23T04:42:48Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.219139] nix-grpc-daemon[773]: ts=2026-09-23T04:42:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.296049] nix-grpc-daemon[770]: ts=2026-09-23T04:42:48Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.301404] nix-grpc-daemon[770]: ts=2026-09-23T04:42:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.420767] nix-grpc-daemon[773]: ts=2026-09-23T04:42:48Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.427091] nix-grpc-daemon[773]: ts=2026-09-23T04:42:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.502311] nix-grpc-daemon[770]: ts=2026-09-23T04:42:49Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.507979] nix-grpc-daemon[770]: ts=2026-09-23T04:42:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.628335] nix-grpc-daemon[773]: ts=2026-09-23T04:42:49Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.634037] nix-grpc-daemon[773]: ts=2026-09-23T04:42:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.710953] nix-grpc-daemon[770]: ts=2026-09-23T04:42:49Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.715391] nix-grpc-daemon[770]: ts=2026-09-23T04:42:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.837225] nix-grpc-daemon[773]: ts=2026-09-23T04:42:49Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.842679] nix-grpc-daemon[773]: ts=2026-09-23T04:42:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.912072] nix-grpc-daemon[770]: ts=2026-09-23T04:42:49Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.917678] nix-grpc-daemon[770]: ts=2026-09-23T04:42:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.044107] nix-grpc-daemon[773]: ts=2026-09-23T04:42:49Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.049500] nix-grpc-daemon[773]: ts=2026-09-23T04:42:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.118641] nix-grpc-daemon[770]: ts=2026-09-23T04:42:49Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.124412] nix-grpc-daemon[770]: ts=2026-09-23T04:42:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.251130] nix-grpc-daemon[773]: ts=2026-09-23T04:42:49Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.256982] nix-grpc-daemon[773]: ts=2026-09-23T04:42:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.325274] nix-grpc-daemon[770]: ts=2026-09-23T04:42:49Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.330958] nix-grpc-daemon[770]: ts=2026-09-23T04:42:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.458596] nix-grpc-daemon[773]: ts=2026-09-23T04:42:49Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.464559] nix-grpc-daemon[773]: ts=2026-09-23T04:42:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.532525] nix-grpc-daemon[770]: ts=2026-09-23T04:42:50Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.539204] nix-grpc-daemon[770]: ts=2026-09-23T04:42:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.667050] nix-grpc-daemon[773]: ts=2026-09-23T04:42:50Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.673863] nix-grpc-daemon[773]: ts=2026-09-23T04:42:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.742898] nix-grpc-daemon[770]: ts=2026-09-23T04:42:50Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.747906] nix-grpc-daemon[770]: ts=2026-09-23T04:42:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.877057] nix-grpc-daemon[773]: ts=2026-09-23T04:42:50Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.881973] nix-grpc-daemon[773]: ts=2026-09-23T04:42:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.944108] nix-grpc-daemon[770]: ts=2026-09-23T04:42:50Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.949790] nix-grpc-daemon[770]: ts=2026-09-23T04:42:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.083527] nix-grpc-daemon[773]: ts=2026-09-23T04:42:50Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.089385] nix-grpc-daemon[773]: ts=2026-09-23T04:42:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.151075] nix-grpc-daemon[770]: ts=2026-09-23T04:42:50Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.164339] nix-grpc-daemon[770]: ts=2026-09-23T04:42:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.290793] nix-grpc-daemon[773]: ts=2026-09-23T04:42:50Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.296969] nix-grpc-daemon[773]: ts=2026-09-23T04:42:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 29.307989] rustfs-bucket-start[992]: mb s3://farm lb # [ 29.316426] systemd[1]: Finished rustfs-bucket.service. worker1 # [ 29.358331] nix-grpc-daemon[770]: ts=2026-09-23T04:42:50Z level=info event=scheduler_connected addr=lb:50051 lb # [ 29.331860] systemd[1]: Starting niks3 server... worker1 # [ 29.364290] nix-grpc-daemon[770]: ts=2026-09-23T04:42:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.498805] nix-grpc-daemon[773]: ts=2026-09-23T04:42:51Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.504470] nix-grpc-daemon[773]: ts=2026-09-23T04:42:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 29.520488] postgres[1007]: [1007] ERROR: relation "goose_db_version" does not exist at character 36 lb # [ 29.521901] postgres[1007]: [1007] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC worker1 # [ 29.565003] nix-grpc-daemon[770]: ts=2026-09-23T04:42:51Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.571397] nix-grpc-daemon[770]: ts=2026-09-23T04:42:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 29.556092] niks3-server[1001]: 2026/09/23 04:42:51 OK 20241026095416_initial_model.sql (22.23ms) lb # [ 29.560246] niks3-server[1001]: 2026/09/23 04:42:51 OK 20251210153512_drop_unused_gin_index.sql (4.48ms) lb # [ 29.566570] niks3-server[1001]: 2026/09/23 04:42:51 OK 20251218171726_add_pins.sql (6.3ms) lb # [ 29.572325] niks3-server[1001]: 2026/09/23 04:42:51 OK 20260628120000_add_object_size_and_stats.sql (5.71ms) lb # [ 29.578597] niks3-server[1001]: 2026/09/23 04:42:51 OK 20260905000000_add_claims.sql (6.22ms) lb # [ 29.583234] niks3-server[1001]: 2026/09/23 04:42:51 OK 20260920000000_drop_claims.sql (4.43ms) lb # [ 29.585500] niks3-server[1001]: 2026/09/23 04:42:51 goose: successfully migrated database to version: 20260920000000 lb # [ 29.589894] niks3-server[1001]: 2026/09/23 04:42:51 OK 1_commit_pending_closure.sql (6.65ms) lb # [ 29.593274] niks3-server[1001]: 2026/09/23 04:42:51 OK 2_object_stats_trigger.sql (3.21ms) lb # [ 29.595471] niks3-server[1001]: 2026/09/23 04:42:51 goose: up to current file version: 2 lb # [ 29.597423] niks3-server[1001]: 2026/09/23 04:42:51 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key lb # [ 29.626013] niks3-server[1001]: 2026/09/23 04:42:51 INFO Read proxy enabled — serving cache objects from S3 lb # [ 29.633436] niks3-server[1001]: 2026/09/23 04:42:51 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 29.640329] niks3-server[1001]: 2026/09/23 04:42:51 INFO systemd watchdog enabled interval=15s lb # [ 29.641689] niks3-server[1001]: 2026/09/23 04:42:51 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 29.643027] systemd[1]: Started niks3 server. lb # [ 29.643710] systemd[1]: Reached target Multi-User System. lb # [ 29.647960] systemd[1]: Startup finished in 1.041s (kernel) + 5.075s (initrd) + 23.520s (userspace) = 29.636s. worker2 # [ 29.701380] nix-grpc-daemon[773]: ts=2026-09-23T04:42:51Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.707442] nix-grpc-daemon[773]: ts=2026-09-23T04:42:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: waiting for unit niks3.service, in 15.88 seconds) lb: waiting for TCP port 5751 on localhost worker1 # [ 29.774228] nix-grpc-daemon[770]: ts=2026-09-23T04:42:51Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.779295] nix-grpc-daemon[770]: ts=2026-09-23T04:42:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded! lb: (finished: waiting for TCP port 5751 on localhost, in 0.04 seconds) lb: waiting for unit envoy.service lb: (finished: waiting for unit envoy.service, in 0.03 seconds) lb: waiting for TCP port 50051 on localhost lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded! lb # [ 29.840617] envoy[891]: conn peer=[::1]:38772 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=3 lb: (finished: waiting for TCP port 50051 on localhost, in 0.03 seconds) worker2 # [ 29.909626] nix-grpc-daemon[773]: ts=2026-09-23T04:42:51Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.912776] nix-grpc-daemon[773]: ts=2026-09-23T04:42:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.980356] nix-grpc-daemon[770]: ts=2026-09-23T04:42:51Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.986161] nix-grpc-daemon[770]: ts=2026-09-23T04:42:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.057718] systemd[1]: Starting scheduler-token.service... worker2 # [ 30.110390] nix-grpc-daemon[773]: ts=2026-09-23T04:42:51Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.113977] nix-grpc-daemon[773]: ts=2026-09-23T04:42:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.123985] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 30.128172] systemd[1]: Finished scheduler-token.service. worker2 # [ 30.129576] systemd[1]: scheduler-token.service: Consumed 33ms CPU time over 69ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 30.187130] nix-grpc-daemon[770]: ts=2026-09-23T04:42:51Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.193047] nix-grpc-daemon[770]: ts=2026-09-23T04:42:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.08 seconds) subtest: one node schedules and both builders hold a WorkerSession on it lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2 # [ 30.315787] nix-grpc-daemon[773]: ts=2026-09-23T04:42:51Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.321790] nix-grpc-daemon[773]: ts=2026-09-23T04:42:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.394112] nix-grpc-daemon[770]: ts=2026-09-23T04:42:51Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.400166] nix-grpc-daemon[770]: ts=2026-09-23T04:42:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.523164] nix-grpc-daemon[773]: ts=2026-09-23T04:42:52Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.529459] nix-grpc-daemon[773]: ts=2026-09-23T04:42:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.601152] nix-grpc-daemon[770]: ts=2026-09-23T04:42:52Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.607442] nix-grpc-daemon[770]: ts=2026-09-23T04:42:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.730785] nix-grpc-daemon[773]: ts=2026-09-23T04:42:52Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.736939] nix-grpc-daemon[773]: ts=2026-09-23T04:42:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.810535] nix-grpc-daemon[770]: ts=2026-09-23T04:42:52Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.815093] nix-grpc-daemon[770]: ts=2026-09-23T04:42:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.940230] nix-grpc-daemon[773]: ts=2026-09-23T04:42:52Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.944602] nix-grpc-daemon[773]: ts=2026-09-23T04:42:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.011394] nix-grpc-daemon[770]: ts=2026-09-23T04:42:52Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.017264] nix-grpc-daemon[770]: ts=2026-09-23T04:42:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.141353] nix-grpc-daemon[773]: ts=2026-09-23T04:42:52Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.146930] nix-grpc-daemon[773]: ts=2026-09-23T04:42:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.218189] nix-grpc-daemon[770]: ts=2026-09-23T04:42:52Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.224128] nix-grpc-daemon[770]: ts=2026-09-23T04:42:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 31.348104] nix-grpc-daemon[773]: ts=2026-09-23T04:42:52Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.353733] nix-grpc-daemon[773]: ts=2026-09-23T04:42:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.425200] nix-grpc-daemon[770]: ts=2026-09-23T04:42:52Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.431149] nix-grpc-daemon[770]: ts=2026-09-23T04:42:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.555272] nix-grpc-daemon[773]: ts=2026-09-23T04:42:53Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.561599] nix-grpc-daemon[773]: ts=2026-09-23T04:42:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.631729] nix-grpc-daemon[770]: ts=2026-09-23T04:42:53Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.635245] nix-grpc-daemon[770]: ts=2026-09-23T04:42:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.762932] nix-grpc-daemon[773]: ts=2026-09-23T04:42:53Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.769346] nix-grpc-daemon[773]: ts=2026-09-23T04:42:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.838421] nix-grpc-daemon[770]: ts=2026-09-23T04:42:53Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.842879] nix-grpc-daemon[770]: ts=2026-09-23T04:42:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.973159] nix-grpc-daemon[773]: ts=2026-09-23T04:42:53Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.977696] nix-grpc-daemon[773]: ts=2026-09-23T04:42:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.039499] nix-grpc-daemon[770]: ts=2026-09-23T04:42:53Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.045522] nix-grpc-daemon[770]: ts=2026-09-23T04:42:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.174067] nix-grpc-daemon[773]: ts=2026-09-23T04:42:53Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.179824] nix-grpc-daemon[773]: ts=2026-09-23T04:42:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.246447] nix-grpc-daemon[770]: ts=2026-09-23T04:42:53Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.252489] nix-grpc-daemon[770]: ts=2026-09-23T04:42:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 32.381615] nix-grpc-daemon[773]: ts=2026-09-23T04:42:53Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.387751] nix-grpc-daemon[773]: ts=2026-09-23T04:42:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1 # [ 32.453444] nix-grpc-daemon[770]: ts=2026-09-23T04:42:53Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.459387] nix-grpc-daemon[770]: ts=2026-09-23T04:42:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.589099] nix-grpc-daemon[773]: ts=2026-09-23T04:42:54Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.594531] nix-grpc-daemon[773]: ts=2026-09-23T04:42:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.660479] nix-grpc-daemon[770]: ts=2026-09-23T04:42:54Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.666598] nix-grpc-daemon[770]: ts=2026-09-23T04:42:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.796070] nix-grpc-daemon[773]: ts=2026-09-23T04:42:54Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.802427] nix-grpc-daemon[773]: ts=2026-09-23T04:42:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.870000] nix-grpc-daemon[770]: ts=2026-09-23T04:42:54Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.874660] nix-grpc-daemon[770]: ts=2026-09-23T04:42:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.005304] nix-grpc-daemon[773]: ts=2026-09-23T04:42:54Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.011045] nix-grpc-daemon[773]: ts=2026-09-23T04:42:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.071210] nix-grpc-daemon[770]: ts=2026-09-23T04:42:54Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.076852] nix-grpc-daemon[770]: ts=2026-09-23T04:42:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.212883] nix-grpc-daemon[773]: ts=2026-09-23T04:42:54Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.218995] nix-grpc-daemon[773]: ts=2026-09-23T04:42:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.277805] nix-grpc-daemon[770]: ts=2026-09-23T04:42:54Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.281684] nix-grpc-daemon[770]: ts=2026-09-23T04:42:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 33.420087] nix-grpc-daemon[773]: ts=2026-09-23T04:42:54Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.425585] nix-grpc-daemon[773]: ts=2026-09-23T04:42:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker1 # [ 33.482847] nix-grpc-daemon[770]: ts=2026-09-23T04:42:55Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.490432] nix-grpc-daemon[770]: ts=2026-09-23T04:42:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.626761] nix-grpc-daemon[773]: ts=2026-09-23T04:42:55Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.632758] nix-grpc-daemon[773]: ts=2026-09-23T04:42:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.689999] nix-grpc-daemon[770]: ts=2026-09-23T04:42:55Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.695555] nix-grpc-daemon[770]: ts=2026-09-23T04:42:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.833716] nix-grpc-daemon[773]: ts=2026-09-23T04:42:55Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.839272] nix-grpc-daemon[773]: ts=2026-09-23T04:42:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.898661] nix-grpc-daemon[770]: ts=2026-09-23T04:42:55Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.903022] nix-grpc-daemon[770]: ts=2026-09-23T04:42:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.042821] nix-grpc-daemon[773]: ts=2026-09-23T04:42:55Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.047280] nix-grpc-daemon[773]: ts=2026-09-23T04:42:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.099301] nix-grpc-daemon[770]: ts=2026-09-23T04:42:55Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.102588] nix-grpc-daemon[770]: ts=2026-09-23T04:42:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.244152] nix-grpc-daemon[773]: ts=2026-09-23T04:42:55Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.249767] nix-grpc-daemon[773]: ts=2026-09-23T04:42:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.303566] nix-grpc-daemon[770]: ts=2026-09-23T04:42:55Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.309160] nix-grpc-daemon[770]: ts=2026-09-23T04:42:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.451006] nix-grpc-daemon[773]: ts=2026-09-23T04:42:55Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.454953] nix-grpc-daemon[773]: ts=2026-09-23T04:42:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 34.509951] nix-grpc-daemon[770]: ts=2026-09-23T04:42:56Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.515671] nix-grpc-daemon[770]: ts=2026-09-23T04:42:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker2 # [ 34.656374] nix-grpc-daemon[773]: ts=2026-09-23T04:42:56Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.661857] nix-grpc-daemon[773]: ts=2026-09-23T04:42:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.716718] nix-grpc-daemon[770]: ts=2026-09-23T04:42:56Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.722442] nix-grpc-daemon[770]: ts=2026-09-23T04:42:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.863375] nix-grpc-daemon[773]: ts=2026-09-23T04:42:56Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.869924] nix-grpc-daemon[773]: ts=2026-09-23T04:42:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.925699] nix-grpc-daemon[770]: ts=2026-09-23T04:42:56Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.930049] nix-grpc-daemon[770]: ts=2026-09-23T04:42:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.073620] nix-grpc-daemon[773]: ts=2026-09-23T04:42:56Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.077881] nix-grpc-daemon[773]: ts=2026-09-23T04:42:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 35.126316] nix-grpc-daemon[770]: ts=2026-09-23T04:42:56Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 35.131863] nix-grpc-daemon[770]: ts=2026-09-23T04:42:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.274720] nix-grpc-daemon[773]: ts=2026-09-23T04:42:56Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.280726] nix-grpc-daemon[773]: ts=2026-09-23T04:42:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 35.332927] nix-grpc-daemon[770]: ts=2026-09-23T04:42:56Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 35.338439] nix-grpc-daemon[770]: ts=2026-09-23T04:42:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.481832] nix-grpc-daemon[773]: ts=2026-09-23T04:42:57Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.487287] nix-grpc-daemon[773]: ts=2026-09-23T04:42:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 35.539254] nix-grpc-daemon[770]: ts=2026-09-23T04:42:57Z level=info event=scheduler_connected addr=lb:50051 lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 35.544985] nix-grpc-daemon[770]: ts=2026-09-23T04:42:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.11 seconds) worker2 # [ 35.688698] nix-grpc-daemon[773]: ts=2026-09-23T04:42:57Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.694278] nix-grpc-daemon[773]: ts=2026-09-23T04:42:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 35.746071] nix-grpc-daemon[770]: ts=2026-09-23T04:42:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 35.751821] nix-grpc-daemon[770]: ts=2026-09-23T04:42:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.895119] nix-grpc-daemon[773]: ts=2026-09-23T04:42:57Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.899095] nix-grpc-daemon[773]: ts=2026-09-23T04:42:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 35.954952] nix-grpc-daemon[770]: ts=2026-09-23T04:42:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 35.959277] nix-grpc-daemon[770]: ts=2026-09-23T04:42:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.102368] nix-grpc-daemon[773]: ts=2026-09-23T04:42:57Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.106811] nix-grpc-daemon[773]: ts=2026-09-23T04:42:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 36.156271] nix-grpc-daemon[770]: ts=2026-09-23T04:42:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.161885] nix-grpc-daemon[770]: ts=2026-09-23T04:42:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.303294] nix-grpc-daemon[773]: ts=2026-09-23T04:42:57Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.306800] nix-grpc-daemon[773]: ts=2026-09-23T04:42:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 36.362991] nix-grpc-daemon[770]: ts=2026-09-23T04:42:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.368709] nix-grpc-daemon[770]: ts=2026-09-23T04:42:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.508409] nix-grpc-daemon[773]: ts=2026-09-23T04:42:58Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.514640] nix-grpc-daemon[773]: ts=2026-09-23T04:42:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 36.569728] nix-grpc-daemon[770]: ts=2026-09-23T04:42:58Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.575566] nix-grpc-daemon[770]: ts=2026-09-23T04:42:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 36.716101] nix-grpc-daemon[773]: ts=2026-09-23T04:42:58Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.721593] nix-grpc-daemon[773]: ts=2026-09-23T04:42:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker1 # [ 36.776629] nix-grpc-daemon[770]: ts=2026-09-23T04:42:58Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.782225] nix-grpc-daemon[770]: ts=2026-09-23T04:42:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.922987] nix-grpc-daemon[773]: ts=2026-09-23T04:42:58Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.928750] nix-grpc-daemon[773]: ts=2026-09-23T04:42:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 36.985394] nix-grpc-daemon[770]: ts=2026-09-23T04:42:58Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.989726] nix-grpc-daemon[770]: ts=2026-09-23T04:42:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.132113] nix-grpc-daemon[773]: ts=2026-09-23T04:42:58Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.136338] nix-grpc-daemon[773]: ts=2026-09-23T04:42:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 37.185888] nix-grpc-daemon[770]: ts=2026-09-23T04:42:58Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.191467] nix-grpc-daemon[770]: ts=2026-09-23T04:42:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.333053] nix-grpc-daemon[773]: ts=2026-09-23T04:42:58Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.338601] nix-grpc-daemon[773]: ts=2026-09-23T04:42:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 37.392526] nix-grpc-daemon[770]: ts=2026-09-23T04:42:58Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.398118] nix-grpc-daemon[770]: ts=2026-09-23T04:42:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.539725] nix-grpc-daemon[773]: ts=2026-09-23T04:42:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.545184] nix-grpc-daemon[773]: ts=2026-09-23T04:42:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 37.598979] nix-grpc-daemon[770]: ts=2026-09-23T04:42:59Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.604751] nix-grpc-daemon[770]: ts=2026-09-23T04:42:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 37.746513] nix-grpc-daemon[773]: ts=2026-09-23T04:42:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.752067] nix-grpc-daemon[773]: ts=2026-09-23T04:42:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 37.805722] nix-grpc-daemon[770]: ts=2026-09-23T04:42:59Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.811322] nix-grpc-daemon[770]: ts=2026-09-23T04:42:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.953672] nix-grpc-daemon[773]: ts=2026-09-23T04:42:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.959501] nix-grpc-daemon[773]: ts=2026-09-23T04:42:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 38.014296] nix-grpc-daemon[770]: ts=2026-09-23T04:42:59Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.018736] nix-grpc-daemon[770]: ts=2026-09-23T04:42:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.162721] nix-grpc-daemon[773]: ts=2026-09-23T04:42:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.167121] nix-grpc-daemon[773]: ts=2026-09-23T04:42:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 38.215101] nix-grpc-daemon[770]: ts=2026-09-23T04:42:59Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.220786] nix-grpc-daemon[770]: ts=2026-09-23T04:42:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.363687] nix-grpc-daemon[773]: ts=2026-09-23T04:42:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.369371] nix-grpc-daemon[773]: ts=2026-09-23T04:42:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 38.421564] nix-grpc-daemon[770]: ts=2026-09-23T04:42:59Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.427085] nix-grpc-daemon[770]: ts=2026-09-23T04:42:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.570554] nix-grpc-daemon[773]: ts=2026-09-23T04:43:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.576257] nix-grpc-daemon[773]: ts=2026-09-23T04:43:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 38.628061] nix-grpc-daemon[770]: ts=2026-09-23T04:43:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.633637] nix-grpc-daemon[770]: ts=2026-09-23T04:43:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.777630] nix-grpc-daemon[773]: ts=2026-09-23T04:43:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.783153] nix-grpc-daemon[773]: ts=2026-09-23T04:43:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 38.834425] nix-grpc-daemon[770]: ts=2026-09-23T04:43:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.840143] nix-grpc-daemon[770]: ts=2026-09-23T04:43:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker2 # [ 38.984390] nix-grpc-daemon[773]: ts=2026-09-23T04:43:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.986909] nix-grpc-daemon[773]: ts=2026-09-23T04:43:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 39.043142] nix-grpc-daemon[770]: ts=2026-09-23T04:43:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.047555] nix-grpc-daemon[770]: ts=2026-09-23T04:43:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 39.190640] nix-grpc-daemon[773]: ts=2026-09-23T04:43:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.195895] nix-grpc-daemon[773]: ts=2026-09-23T04:43:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 39.243644] nix-grpc-daemon[770]: ts=2026-09-23T04:43:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.248387] nix-grpc-daemon[770]: ts=2026-09-23T04:43:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 39.391612] nix-grpc-daemon[773]: ts=2026-09-23T04:43:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.399987] nix-grpc-daemon[773]: ts=2026-09-23T04:43:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 39.449453] nix-grpc-daemon[770]: ts=2026-09-23T04:43:00Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.455306] nix-grpc-daemon[770]: ts=2026-09-23T04:43:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 39.601244] nix-grpc-daemon[773]: ts=2026-09-23T04:43:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.608279] nix-grpc-daemon[773]: ts=2026-09-23T04:43:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 39.656006] nix-grpc-daemon[770]: ts=2026-09-23T04:43:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.661549] nix-grpc-daemon[770]: ts=2026-09-23T04:43:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 39.668118] nix-grpc-daemon[770]: ts=2026-09-23T04:43:01Z level=info event=scheduler_take_over lb # [ 39.637419] niks3-server[1001]: 2026/09/23 04:43:01 INFO lead: acquired remote=192.168.1.3:38116 worker2 # [ 39.809368] nix-grpc-daemon[773]: ts=2026-09-23T04:43:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.814993] nix-grpc-daemon[773]: ts=2026-09-23T04:43:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 39.862554] nix-grpc-daemon[770]: ts=2026-09-23T04:43:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.868216] nix-grpc-daemon[770]: ts=2026-09-23T04:43:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker2 # [ 40.016364] nix-grpc-daemon[773]: ts=2026-09-23T04:43:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 40.021982] nix-grpc-daemon[773]: ts=2026-09-23T04:43:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 40.071232] nix-grpc-daemon[770]: ts=2026-09-23T04:43:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 40.075710] nix-grpc-daemon[770]: ts=2026-09-23T04:43:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 40.225117] nix-grpc-daemon[773]: ts=2026-09-23T04:43:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 40.230046] nix-grpc-daemon[773]: ts=2026-09-23T04:43:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 40.271491] nix-grpc-daemon[770]: ts=2026-09-23T04:43:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 40.274934] nix-grpc-daemon[770]: ts=2026-09-23T04:43:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 40.431468] nix-grpc-daemon[773]: ts=2026-09-23T04:43:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 40.437261] nix-grpc-daemon[773]: ts=2026-09-23T04:43:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 40.475927] nix-grpc-daemon[770]: ts=2026-09-23T04:43:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 40.480875] nix-grpc-daemon[770]: ts=2026-09-23T04:43:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 40.638218] nix-grpc-daemon[773]: ts=2026-09-23T04:43:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 40.654168] nix-grpc-daemon[770]: ts=2026-09-23T04:43:02Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 40.658946] nix-grpc-daemon[770]: ts=2026-09-23T04:43:02Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:45280 worker1 # [ 40.664412] nix-grpc-daemon[770]: ts=2026-09-23T04:43:02Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 40.681723] nix-grpc-daemon[770]: ts=2026-09-23T04:43:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 40.687714] nix-grpc-daemon[770]: ts=2026-09-23T04:43:02Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:45280 worker1 # [ 40.692976] nix-grpc-daemon[770]: ts=2026-09-23T04:43:02Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open' worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.06 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 11.53 seconds) subtest: balancer auth: client cert, bearer token, nothing, unknown CN worker2 # [ 42.034224] nix-grpc-daemon[773]: ts=2026-09-23T04:43:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 42.051982] systemd[1]: Started Nix Daemon. worker2 # [ 42.243809] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 42.266440] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 26 ms (attempt 1/5) worker2 # [ 42.296276] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 104 ms (attempt 2/5) worker2 # [ 42.405678] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 183 ms (attempt 3/5) worker2 # [ 42.591131] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 35 ms (attempt 4/5) worker2 # [ 42.628990] nix-grpc-daemon[773]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 42.633923] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 42.651613] nix-grpc-daemon[770]: ts=2026-09-23T04:43:04Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 42.670041] systemd[1]: Started Nix Daemon. worker1 # [ 42.854107] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 42.872777] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 13 ms (attempt 1/5) worker1 # [ 42.889668] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 184 ms (attempt 2/5) worker1 # [ 43.078493] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 97 ms (attempt 3/5) worker1 # [ 43.177660] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 575 ms (attempt 4/5) worker1 # [ 43.757159] nix-grpc-daemon[770]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 43.762524] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 43.773329] nix-grpc-daemon[770]: ts=2026-09-23T04:43:05Z level=debug event=path_info_miss path=h0h42468m1lycnnbixi8y4srnpd3qprr-nixos-system-client-test error="error: path '/nix/store/h0h42468m1lycnnbixi8y4srnpd3qprr-nixos-system-client-test' is required, but there is no substituter that can build it" worker1 # [ 43.789991] nix-grpc-daemon[770]: ts=2026-09-23T04:43:05Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 43.798325] nix-grpc-daemon[770]: ts=2026-09-23T04:43:05Z level=debug event=path_info_miss path=h0h42468m1lycnnbixi8y4srnpd3qprr-nixos-system-client-test error="error: path '/nix/store/h0h42468m1lycnnbixi8y4srnpd3qprr-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.09 seconds) client: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key client: (finished: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds) worker1 # [ 44.168228] nix-grpc-daemon[770]: ts=2026-09-23T04:43:05Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:59106 worker1 # [ 44.182089] nix-grpc-daemon[770]: ts=2026-09-23T04:43:05Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:59106 worker1 # [ 44.195184] nix-grpc-daemon[770]: ts=2026-09-23T04:43:05Z level=debug event=path_info_miss path=h0h42468m1lycnnbixi8y4srnpd3qprr-nixos-system-client-test error="error: path '/nix/store/h0h42468m1lycnnbixi8y4srnpd3qprr-nixos-system-client-test' is required, but there is no substituter that can build it" worker2 # [ 44.223897] nix-grpc-daemon[773]: ts=2026-09-23T04:43:05Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 44.231696] nix-grpc-daemon[773]: ts=2026-09-23T04:43:05Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:32768 worker2 # [ 44.245727] nix-grpc-daemon[773]: ts=2026-09-23T04:43:05Z level=debug event=path_info_miss path=h0h42468m1lycnnbixi8y4srnpd3qprr-nixos-system-client-test error="error: path '/nix/store/h0h42468m1lycnnbixi8y4srnpd3qprr-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: rm -r /var/lib/nix-grpc-store client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.02 seconds) worker1 # [ 44.429229] nix-grpc-daemon[770]: ts=2026-09-23T04:43:05Z level=info event=denied method=QueryMissing cn=- role=none worker1 # [ 44.615338] nix-grpc-daemon[770]: ts=2026-09-23T04:43:06Z level=info event=denied method=QueryMissing cn=stranger role=none (finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 2.89 seconds) subtest: DAG build is scheduled across workers and lands in the cache client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 worker2 # [ 44.812926] nix-grpc-daemon[773]: ts=2026-09-23T04:43:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:32768 client # don't know how to build these paths: client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker1 # [ 44.813510] nix-grpc-daemon[770]: ts=2026-09-23T04:43:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 44.856181] nix-grpc-daemon[770]: ts=2026-09-23T04:43:06Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=0 worker1 # [ 44.862805] nix-grpc-daemon[770]: ts=2026-09-23T04:43:06Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 44.929175] nix-grpc-daemon[773]: ts=2026-09-23T04:43:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 44.947148] nix-grpc-daemon[773]: ts=2026-09-23T04:43:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'... worker2 # [ 44.964949] nix-grpc-daemon[773]: ts=2026-09-23T04:43:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 44.986050] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 45.023983] nix-grpc-daemon[773]: ts=2026-09-23T04:43:06Z level=info event=niks3_push_started pid=856 worker2 # [ 45.240496] nix-grpc-daemon[856]: time=2026-09-23T04:43:06.757Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 45.358970] nix-daemon[843]: accepted connection from pid 869, user nix-grpc-daemon (trusted) worker2 # [ 45.383545] nix-daemon[843]: reaped child process 872, status = succeeded lb # [ 45.341461] niks3-server[1001]: 2026/09/23 04:43:06 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 45.410079] nix-grpc-daemon[856]: time=2026-09-23T04:43:06.927Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 45.411693] nix-grpc-daemon[856]: time=2026-09-23T04:43:06.927Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)" lb # [ 45.406802] niks3-server[1001]: 2026/09/23 04:43:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign lb # [ 45.427561] niks3-server[1001]: 2026/09/23 04:43:06 INFO Signed narinfos id=1 count=1 worker2 # [ 45.476146] nix-grpc-daemon[856]: time=2026-09-23T04:43:06.993Z level=INFO msg="Uploading 1 narinfos" lb # [ 45.454687] niks3-server[1001]: 2026/09/23 04:43:06 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst lb # [ 45.463726] niks3-server[1001]: 2026/09/23 04:43:06 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls lb # [ 45.471218] niks3-server[1001]: 2026/09/23 04:43:06 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete lb # [ 45.485467] niks3-server[1001]: 2026/09/23 04:43:07 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo lb # [ 45.490588] niks3-server[1001]: 2026/09/23 04:43:07 INFO Completed upload id=1 worker2 # [ 45.534119] nix-grpc-daemon[856]: time=2026-09-23T04:43:07.052Z level=INFO msg="Upload complete. (293ms)" worker2 # [ 45.549386] nix-grpc-daemon[773]: ts=2026-09-23T04:43:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_in=504 worker2 # [ 45.569411] nix-grpc-daemon[773]: ts=2026-09-23T04:43:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 45.577447] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 45.583395] nix-daemon[843]: reaped child process 854, status = succeeded client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051... worker2 # [ 45.597819] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) client # farm-a> worker2: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-t1 worker2 # [ 46.631163] nix-daemon[843]: reaped child process 876, status = succeeded worker2 # [ 46.738282] nix-daemon[843]: accepted connection from pid 893, user nix-grpc-daemon (trusted) worker2 # [ 46.761962] nix-daemon[843]: reaped child process 896, status = succeeded lb # [ 46.719387] niks3-server[1001]: 2026/09/23 04:43:08 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 46.779759] nix-grpc-daemon[856]: time=2026-09-23T04:43:08.295Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 46.782114] nix-grpc-daemon[856]: time=2026-09-23T04:43:08.295Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)" lb # [ 46.768822] niks3-server[1001]: 2026/09/23 04:43:08 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv lb # [ 46.789547] niks3-server[1001]: 2026/09/23 04:43:08 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls lb # [ 46.799086] niks3-server[1001]: 2026/09/23 04:43:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign worker2 # [ 46.844516] nix-grpc-daemon[856]: time=2026-09-23T04:43:08.362Z level=INFO msg="Uploading 1 narinfos" lb # [ 46.808954] niks3-server[1001]: 2026/09/23 04:43:08 INFO Signed narinfos id=2 count=1 lb # [ 46.814627] niks3-server[1001]: 2026/09/23 04:43:08 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst lb # [ 46.827008] niks3-server[1001]: 2026/09/23 04:43:08 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete lb # [ 46.834848] niks3-server[1001]: 2026/09/23 04:43:08 INFO Completed upload id=2 worker2 # [ 46.878851] nix-grpc-daemon[856]: time=2026-09-23T04:43:08.397Z level=INFO msg="Upload complete. (265ms)" lb # [ 46.838082] niks3-server[1001]: 2026/09/23 04:43:08 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo worker1 # [ 46.891953] nix-grpc-daemon[770]: ts=2026-09-23T04:43:08Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=0 worker2 # [ 46.893344] nix-grpc-daemon[773]: ts=2026-09-23T04:43:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=1 worker1 # [ 46.892874] nix-grpc-daemon[770]: ts=2026-09-23T04:43:08Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 46.915361] nix-daemon[843]: reaped child process 874, status = succeeded worker2 # [ 46.950204] nix-grpc-daemon[773]: ts=2026-09-23T04:43:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 46.968344] nix-grpc-daemon[773]: ts=2026-09-23T04:43:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 46.985425] nix-grpc-daemon[856]: time=2026-09-23T04:43:08.503Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 46.998442] nix-grpc-daemon[773]: ts=2026-09-23T04:43:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'... worker2 # [ 47.023521] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 47.203835] nix-daemon[843]: accepted connection from pid 900, user nix-grpc-daemon (trusted) worker2 # [ 47.229133] nix-daemon[843]: reaped child process 903, status = succeeded lb # [ 47.188535] niks3-server[1001]: 2026/09/23 04:43:08 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 47.244110] nix-grpc-daemon[856]: time=2026-09-23T04:43:08.762Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 47.245972] nix-grpc-daemon[856]: time=2026-09-23T04:43:08.762Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)" lb # [ 47.220883] niks3-server[1001]: 2026/09/23 04:43:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign worker2 # [ 47.272777] nix-grpc-daemon[856]: time=2026-09-23T04:43:08.791Z level=INFO msg="Uploading 1 narinfos" lb # [ 47.238128] niks3-server[1001]: 2026/09/23 04:43:08 INFO Signed narinfos id=3 count=1 lb # [ 47.246082] niks3-server[1001]: 2026/09/23 04:43:08 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst lb # [ 47.261265] niks3-server[1001]: 2026/09/23 04:43:08 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls worker2 # [ 47.313863] nix-grpc-daemon[856]: time=2026-09-23T04:43:08.832Z level=INFO msg="Upload complete. (270ms)" lb # [ 47.270582] niks3-server[1001]: 2026/09/23 04:43:08 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete lb # [ 47.277270] niks3-server[1001]: 2026/09/23 04:43:08 INFO Completed upload id=3 worker2 # [ 47.322239] nix-grpc-daemon[773]: ts=2026-09-23T04:43:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 47.281165] niks3-server[1001]: 2026/09/23 04:43:08 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo worker2 # [ 47.336968] nix-grpc-daemon[773]: ts=2026-09-23T04:43:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 47.347185] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 47.357100] nix-daemon[843]: reaped child process 898, status = succeeded client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051... client # farm-b> worker2: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker2 # [ 47.370839] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-t1 worker2 # [ 48.693743] nix-daemon[843]: reaped child process 907, status = succeeded worker2 # [ 48.766133] nix-daemon[843]: accepted connection from pid 923, user nix-grpc-daemon (trusted) worker2 # [ 48.789994] nix-daemon[843]: reaped child process 926, status = succeeded lb # [ 48.753564] niks3-server[1001]: 2026/09/23 04:43:10 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 48.813341] nix-grpc-daemon[856]: time=2026-09-23T04:43:10.330Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 48.815269] nix-grpc-daemon[856]: time=2026-09-23T04:43:10.330Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)" lb # [ 48.813945] niks3-server[1001]: 2026/09/23 04:43:10 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst worker2 # [ 48.865787] nix-grpc-daemon[856]: time=2026-09-23T04:43:10.383Z level=INFO msg="Uploading 1 narinfos" lb # [ 48.828170] niks3-server[1001]: 2026/09/23 04:43:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign lb # [ 48.831879] niks3-server[1001]: 2026/09/23 04:43:10 INFO Signed narinfos id=4 count=1 lb # [ 48.842618] niks3-server[1001]: 2026/09/23 04:43:10 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv lb # [ 48.849606] niks3-server[1001]: 2026/09/23 04:43:10 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls lb # [ 48.861824] niks3-server[1001]: 2026/09/23 04:43:10 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete lb # [ 48.873371] niks3-server[1001]: 2026/09/23 04:43:10 INFO Completed upload id=4 worker2 # [ 48.917882] nix-grpc-daemon[856]: time=2026-09-23T04:43:10.436Z level=INFO msg="Upload complete. (334ms)" lb # [ 48.882972] niks3-server[1001]: 2026/09/23 04:43:10 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo worker2 # [ 48.933322] nix-grpc-daemon[773]: ts=2026-09-23T04:43:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=1 worker1 # [ 48.929172] nix-grpc-daemon[770]: ts=2026-09-23T04:43:10Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=0 worker1 # [ 48.935000] nix-grpc-daemon[770]: ts=2026-09-23T04:43:10Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 48.954625] nix-daemon[843]: reaped child process 905, status = succeeded worker2 # [ 48.986370] nix-grpc-daemon[773]: ts=2026-09-23T04:43:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 49.004418] nix-grpc-daemon[773]: ts=2026-09-23T04:43:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 49.021889] nix-grpc-daemon[856]: time=2026-09-23T04:43:10.540Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'... worker2 # [ 49.034874] nix-grpc-daemon[773]: ts=2026-09-23T04:43:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 49.054028] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 49.219770] nix-daemon[843]: accepted connection from pid 930, user nix-grpc-daemon (trusted) worker2 # [ 49.242585] nix-daemon[843]: reaped child process 933, status = succeeded lb # [ 49.203904] niks3-server[1001]: 2026/09/23 04:43:10 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 49.258893] nix-grpc-daemon[856]: time=2026-09-23T04:43:10.777Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 49.260578] nix-grpc-daemon[856]: time=2026-09-23T04:43:10.777Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)" lb # [ 49.236622] niks3-server[1001]: 2026/09/23 04:43:10 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls worker2 # [ 49.294778] nix-grpc-daemon[856]: time=2026-09-23T04:43:10.813Z level=INFO msg="Uploading 1 narinfos" lb # [ 49.252393] niks3-server[1001]: 2026/09/23 04:43:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign lb # [ 49.265147] niks3-server[1001]: 2026/09/23 04:43:10 INFO Signed narinfos id=5 count=1 lb # [ 49.272506] niks3-server[1001]: 2026/09/23 04:43:10 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst worker2 # [ 49.331971] nix-grpc-daemon[856]: time=2026-09-23T04:43:10.850Z level=INFO msg="Upload complete. (258ms)" lb # [ 49.287878] niks3-server[1001]: 2026/09/23 04:43:10 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete lb # [ 49.291924] niks3-server[1001]: 2026/09/23 04:43:10 INFO Completed upload id=5 lb # [ 49.296494] niks3-server[1001]: 2026/09/23 04:43:10 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo worker2 # [ 49.344185] nix-grpc-daemon[773]: ts=2026-09-23T04:43:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_in=752 worker2 # [ 49.356532] nix-grpc-daemon[773]: ts=2026-09-23T04:43:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 49.371875] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 49.378281] nix-daemon[843]: reaped child process 928, status = succeeded client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051... client # farm-top> worker2: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker2 # [ 49.394668] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds lb # [ 50.181668] envoy[891]: conn peer=[2001:db8:1::1]:60082 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=407 ms=1806 client # farm-top> LOG-farm-top-t1 worker2 # [ 50.436092] nix-daemon[843]: reaped child process 937, status = succeeded worker2 # [ 50.532362] nix-daemon[843]: accepted connection from pid 953, user nix-grpc-daemon (trusted) worker2 # [ 50.554682] nix-daemon[843]: reaped child process 956, status = succeeded lb # [ 50.186834] envoy[891]: conn peer=[2001:db8:1::1]:60096 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=81 lb # [ 50.186986] envoy[891]: conn peer=[2001:db8:1::1]:60098 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=17 lb # [ 50.187090] envoy[891]: conn peer=[2001:db8:1::1]:60112 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=14 lb # [ 50.519718] niks3-server[1001]: 2026/09/23 04:43:12 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 50.592298] nix-grpc-daemon[856]: time=2026-09-23T04:43:12.108Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 50.593954] nix-grpc-daemon[856]: time=2026-09-23T04:43:12.108Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)" lb # [ 50.587081] niks3-server[1001]: 2026/09/23 04:43:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign worker2 # [ 50.644241] nix-grpc-daemon[856]: time=2026-09-23T04:43:12.162Z level=INFO msg="Uploading 1 narinfos" lb # [ 50.607122] niks3-server[1001]: 2026/09/23 04:43:12 INFO Signed narinfos id=6 count=1 lb # [ 50.614359] niks3-server[1001]: 2026/09/23 04:43:12 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst lb # [ 50.624978] niks3-server[1001]: 2026/09/23 04:43:12 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls lb # [ 50.636118] niks3-server[1001]: 2026/09/23 04:43:12 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete lb # [ 50.645257] niks3-server[1001]: 2026/09/23 04:43:12 INFO Completed upload id=6 worker2 # [ 50.689093] nix-grpc-daemon[856]: time=2026-09-23T04:43:12.207Z level=INFO msg="Upload complete. (282ms)" lb # [ 50.653198] niks3-server[1001]: 2026/09/23 04:43:12 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo lb # [ 50.659067] niks3-server[1001]: 2026/09/23 04:43:12 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker2 # [ 50.701451] nix-grpc-daemon[773]: ts=2026-09-23T04:43:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=1 worker2 # [ 50.725420] nix-daemon[843]: reaped child process 935, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 6.11 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.20 seconds) client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.03 seconds) client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client # copying 3 paths... client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'... client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'... client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.24 seconds) (finished: subtest: DAG build is scheduled across workers and lands in the cache, in 6.68 seconds) subtest: repeat is answered Cached by the scheduler without building worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker1 # finding garbage collector roots... worker1 # deleting specified paths... worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker1 # deleting unused links... worker1 # note: hard linking is currently saving 0.0 KiB worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.24 seconds) worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker2 # finding garbage collector roots... worker2 # deleting specified paths... worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker2 # deleting unused links... worker2 # note: hard linking is currently saving 0.0 KiB worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.23 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 worker2 # [ 52.013342] nix-grpc-daemon[773]: ts=2026-09-23T04:43:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:32768 client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked): client # /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker2 # [ 52.033458] nix-grpc-daemon[773]: ts=2026-09-23T04:43:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 52.049143] nix-grpc-daemon[856]: time=2026-09-23T04:43:13.566Z level=INFO msg="All 2 paths already cached" worker1 # [ 52.058358] nix-grpc-daemon[770]: ts=2026-09-23T04:43:13Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=1 client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 0.32 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.17 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: repeat is answered Cached by the scheduler without building, in 1.23 seconds) subtest: inputs already in the cache are not sent to the scheduler lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.04 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2 worker1 # [ 52.917599] nix-grpc-daemon[770]: ts=2026-09-23T04:43:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59106 client # don't know how to build these paths: client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker2 # [ 52.943420] nix-grpc-daemon[773]: ts=2026-09-23T04:43:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 52.953653] nix-grpc-daemon[856]: time=2026-09-23T04:43:14.472Z level=INFO msg="All 2 paths already cached" worker1 # [ 52.959051] nix-grpc-daemon[770]: ts=2026-09-23T04:43:14Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=0 worker1 # [ 52.965428] nix-grpc-daemon[770]: ts=2026-09-23T04:43:14Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 53.018573] nix-grpc-daemon[773]: ts=2026-09-23T04:43:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 53.034786] nix-grpc-daemon[773]: ts=2026-09-23T04:43:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 53.051513] nix-grpc-daemon[856]: time=2026-09-23T04:43:14.569Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'... worker2 # [ 53.064132] nix-grpc-daemon[773]: ts=2026-09-23T04:43:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 53.079608] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 53.242602] nix-daemon[843]: accepted connection from pid 973, user nix-grpc-daemon (trusted) worker2 # [ 53.263587] nix-daemon[843]: reaped child process 976, status = succeeded lb # [ 53.230561] niks3-server[1001]: 2026/09/23 04:43:14 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 53.286936] nix-grpc-daemon[856]: time=2026-09-23T04:43:14.805Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 53.288747] nix-grpc-daemon[856]: time=2026-09-23T04:43:14.805Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)" lb # [ 53.266414] niks3-server[1001]: 2026/09/23 04:43:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign worker2 # [ 53.319263] nix-grpc-daemon[856]: time=2026-09-23T04:43:14.837Z level=INFO msg="Uploading 1 narinfos" lb # [ 53.285012] niks3-server[1001]: 2026/09/23 04:43:14 INFO Signed narinfos id=7 count=1 lb # [ 53.291893] niks3-server[1001]: 2026/09/23 04:43:14 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst lb # [ 53.300938] niks3-server[1001]: 2026/09/23 04:43:14 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls worker2 # [ 53.357537] nix-grpc-daemon[856]: time=2026-09-23T04:43:14.876Z level=INFO msg="Upload complete. (259ms)" lb # [ 53.314553] niks3-server[1001]: 2026/09/23 04:43:14 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete lb # [ 53.323926] niks3-server[1001]: 2026/09/23 04:43:14 INFO Completed upload id=7 worker2 # [ 53.367422] nix-grpc-daemon[773]: ts=2026-09-23T04:43:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 53.327168] niks3-server[1001]: 2026/09/23 04:43:14 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo worker2 # [ 53.380941] nix-grpc-daemon[773]: ts=2026-09-23T04:43:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 53.394508] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 53.404300] nix-daemon[843]: reaped child process 971, status = succeeded client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051... worker2 # [ 53.412170] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) client # farm-top> worker2: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1b worker2 # [ 54.877182] nix-daemon[843]: reaped child process 980, status = succeeded worker2 # [ 55.000073] nix-daemon[843]: accepted connection from pid 996, user nix-grpc-daemon (trusted) worker2 # [ 55.025356] nix-daemon[843]: reaped child process 999, status = succeeded lb # [ 54.979148] niks3-server[1001]: 2026/09/23 04:43:16 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 55.036420] nix-grpc-daemon[856]: time=2026-09-23T04:43:16.552Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 55.037912] nix-grpc-daemon[856]: time=2026-09-23T04:43:16.553Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)" lb # [ 55.022105] niks3-server[1001]: 2026/09/23 04:43:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign worker2 # [ 55.078198] nix-grpc-daemon[856]: time=2026-09-23T04:43:16.596Z level=INFO msg="Uploading 1 narinfos" lb # [ 55.036727] niks3-server[1001]: 2026/09/23 04:43:16 INFO Signed narinfos id=8 count=1 lb # [ 55.039622] niks3-server[1001]: 2026/09/23 04:43:16 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls lb # [ 55.054465] niks3-server[1001]: 2026/09/23 04:43:16 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst lb # [ 55.059414] niks3-server[1001]: 2026/09/23 04:43:16 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv lb # [ 55.068935] niks3-server[1001]: 2026/09/23 04:43:16 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete worker2 # [ 55.112873] nix-grpc-daemon[856]: time=2026-09-23T04:43:16.631Z level=INFO msg="Upload complete. (287ms)" lb # [ 55.072675] niks3-server[1001]: 2026/09/23 04:43:16 INFO Completed upload id=8 lb # [ 55.076789] niks3-server[1001]: 2026/09/23 04:43:16 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo worker2 # [ 55.125358] nix-grpc-daemon[773]: ts=2026-09-23T04:43:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=1 worker2 # [ 55.146270] nix-daemon[843]: reaped child process 978, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.47 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.19 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds) (finished: subtest: inputs already in the cache are not sent to the scheduler, in 3.00 seconds) subtest: outputs read back through any worker carry the cache signature client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2 client # don't know how to build these paths: client # /nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker2 # [ 55.756269] nix-grpc-daemon[773]: ts=2026-09-23T04:43:17Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:32768 worker1 # [ 55.750253] nix-grpc-daemon[770]: ts=2026-09-23T04:43:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 55.775907] nix-grpc-daemon[770]: ts=2026-09-23T04:43:17Z level=debug event=want client=4 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv cached=0 worker1 # [ 55.784502] nix-grpc-daemon[770]: ts=2026-09-23T04:43:17Z level=debug event=assigned drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 55.790993] nix-grpc-daemon[770]: ts=2026-09-23T04:43:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 55.807322] nix-grpc-daemon[770]: ts=2026-09-23T04:43:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59106 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' to 'lb:50051'... worker1 # [ 55.824666] nix-grpc-daemon[770]: ts=2026-09-23T04:43:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 55.840510] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 55.868958] nix-grpc-daemon[770]: ts=2026-09-23T04:43:17Z level=info event=niks3_push_started pid=906 worker1 # [ 56.041838] nix-grpc-daemon[906]: time=2026-09-23T04:43:17.560Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 56.156751] nix-daemon[855]: accepted connection from pid 919, user nix-grpc-daemon (trusted) worker1 # [ 56.182080] nix-daemon[855]: reaped child process 922, status = succeeded lb # [ 56.148926] niks3-server[1001]: 2026/09/23 04:43:17 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 56.194069] nix-grpc-daemon[906]: time=2026-09-23T04:43:17.712Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 56.195619] nix-grpc-daemon[906]: time=2026-09-23T04:43:17.712Z level=INFO msg="Uploading 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv (512B)" lb # [ 56.198643] niks3-server[1001]: 2026/09/23 04:43:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign worker1 # [ 56.242500] nix-grpc-daemon[906]: time=2026-09-23T04:43:17.761Z level=INFO msg="Uploading 1 narinfos" lb # [ 56.213938] niks3-server[1001]: 2026/09/23 04:43:17 INFO Signed narinfos id=9 count=1 lb # [ 56.219272] niks3-server[1001]: 2026/09/23 04:43:17 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.ls lb # [ 56.227375] niks3-server[1001]: 2026/09/23 04:43:17 INFO Registered completed upload object_key=nar/13mhmmy2qay13imbbdi287vhryz1hwnyg0w6agd9s0ph37wnsqll.nar.zst lb # [ 56.237104] niks3-server[1001]: 2026/09/23 04:43:17 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete lb # [ 56.241953] niks3-server[1001]: 2026/09/23 04:43:17 INFO Completed upload id=9 worker1 # [ 56.273964] nix-grpc-daemon[906]: time=2026-09-23T04:43:17.793Z level=INFO msg="Upload complete. (228ms)" lb # [ 56.244572] niks3-server[1001]: 2026/09/23 04:43:17 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.narinfo worker1 # [ 56.284401] nix-grpc-daemon[770]: ts=2026-09-23T04:43:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 56.295470] nix-grpc-daemon[770]: ts=2026-09-23T04:43:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 56.303597] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 56.309622] nix-daemon[855]: reaped child process 904, status = succeeded client # building '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' on lb:50051... client # farm-a> node-a: building 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker1 # [ 56.324167] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-sig worker1 # [ 57.608532] nix-daemon[855]: reaped child process 926, status = succeeded worker1 # [ 57.673132] nix-daemon[855]: accepted connection from pid 942, user nix-grpc-daemon (trusted) worker1 # [ 57.693999] nix-daemon[855]: reaped child process 945, status = succeeded lb # [ 57.667031] niks3-server[1001]: 2026/09/23 04:43:19 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 57.711591] nix-grpc-daemon[906]: time=2026-09-23T04:43:19.228Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 57.713836] nix-grpc-daemon[906]: time=2026-09-23T04:43:19.228Z level=INFO msg="Uploading 2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a (128B)" lb # [ 57.703417] niks3-server[1001]: 2026/09/23 04:43:19 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst lb # [ 57.722068] niks3-server[1001]: 2026/09/23 04:43:19 INFO Registered completed upload object_key=log/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker1 # [ 57.762568] nix-grpc-daemon[906]: time=2026-09-23T04:43:19.281Z level=INFO msg="Uploading 1 narinfos" lb # [ 57.734430] niks3-server[1001]: 2026/09/23 04:43:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign lb # [ 57.741693] niks3-server[1001]: 2026/09/23 04:43:19 INFO Signed narinfos id=10 count=1 lb # [ 57.743736] niks3-server[1001]: 2026/09/23 04:43:19 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.ls worker1 # [ 57.786717] nix-grpc-daemon[906]: time=2026-09-23T04:43:19.306Z level=INFO msg="Upload complete. (251ms)" lb # [ 57.754953] niks3-server[1001]: 2026/09/23 04:43:19 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete lb # [ 57.759822] niks3-server[1001]: 2026/09/23 04:43:19 INFO Completed upload id=10 lb # [ 57.762188] niks3-server[1001]: 2026/09/23 04:43:19 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.narinfo worker1 # [ 57.796750] nix-grpc-daemon[770]: ts=2026-09-23T04:43:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=1 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv assign_id=5 outputs=1 worker1 # [ 57.805637] nix-daemon[855]: reaped child process 924, status = succeeded worker1 # [ 57.807862] nix-grpc-daemon[770]: ts=2026-09-23T04:43:19Z level=debug event=want client=4 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv cached=0 worker1 # [ 57.810503] nix-grpc-daemon[770]: ts=2026-09-23T04:43:19Z level=debug event=assigned drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 57.853626] nix-grpc-daemon[770]: ts=2026-09-23T04:43:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 57.869995] nix-grpc-daemon[770]: ts=2026-09-23T04:43:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 57.884457] nix-grpc-daemon[906]: time=2026-09-23T04:43:19.403Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' to 'lb:50051'... worker1 # [ 57.895948] nix-grpc-daemon[770]: ts=2026-09-23T04:43:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 57.910923] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 58.060985] nix-daemon[855]: accepted connection from pid 951, user nix-grpc-daemon (trusted) worker1 # [ 58.084962] nix-daemon[855]: reaped child process 954, status = succeeded lb # [ 58.056981] niks3-server[1001]: 2026/09/23 04:43:19 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 58.097176] nix-grpc-daemon[906]: time=2026-09-23T04:43:19.616Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 58.098746] nix-grpc-daemon[906]: time=2026-09-23T04:43:19.616Z level=INFO msg="Uploading z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv (624B)" lb # [ 58.078747] niks3-server[1001]: 2026/09/23 04:43:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign worker1 # [ 58.118031] nix-grpc-daemon[906]: time=2026-09-23T04:43:19.637Z level=INFO msg="Uploading 1 narinfos" lb # [ 58.088892] niks3-server[1001]: 2026/09/23 04:43:19 INFO Signed narinfos id=11 count=1 lb # [ 58.094128] niks3-server[1001]: 2026/09/23 04:43:19 INFO Registered completed upload object_key=nar/0cnhkgsbmxv5hj7zw3fa59qfxikw6251mlhz2gz1v6pgn8c8shwp.nar.zst lb # [ 58.102568] niks3-server[1001]: 2026/09/23 04:43:19 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.ls worker1 # [ 58.140232] nix-grpc-daemon[906]: time=2026-09-23T04:43:19.659Z level=INFO msg="Upload complete. (214ms)" lb # [ 58.107911] niks3-server[1001]: 2026/09/23 04:43:19 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete lb # [ 58.111015] niks3-server[1001]: 2026/09/23 04:43:19 INFO Completed upload id=11 worker1 # [ 58.146330] nix-grpc-daemon[770]: ts=2026-09-23T04:43:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 58.114716] niks3-server[1001]: 2026/09/23 04:43:19 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.narinfo worker1 # [ 58.159825] nix-grpc-daemon[770]: ts=2026-09-23T04:43:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 58.166077] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 58.170719] nix-daemon[855]: reaped child process 949, status = succeeded client # building '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' on lb:50051... worker1 # [ 58.181236] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) client # farm-b> node-a: building z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-sig worker1 # [ 59.485500] nix-daemon[855]: reaped child process 958, status = succeeded worker1 # [ 59.559093] nix-daemon[855]: accepted connection from pid 974, user nix-grpc-daemon (trusted) worker1 # [ 59.583093] nix-daemon[855]: reaped child process 977, status = succeeded lb # [ 59.554901] niks3-server[1001]: 2026/09/23 04:43:21 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 59.598392] nix-grpc-daemon[906]: time=2026-09-23T04:43:21.116Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 59.599885] nix-grpc-daemon[906]: time=2026-09-23T04:43:21.116Z level=INFO msg="Uploading h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b (176B)" lb # [ 59.595712] niks3-server[1001]: 2026/09/23 04:43:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign worker1 # [ 59.638703] nix-grpc-daemon[906]: time=2026-09-23T04:43:21.157Z level=INFO msg="Uploading 1 narinfos" lb # [ 59.619526] niks3-server[1001]: 2026/09/23 04:43:21 INFO Signed narinfos id=12 count=1 lb # [ 59.627947] niks3-server[1001]: 2026/09/23 04:43:21 INFO Registered completed upload object_key=nar/0f75gpfkjj0g9xi2113yvvpdw0l16spc2f1xik1ifnzfbrvx4pfk.nar.zst lb # [ 59.645567] niks3-server[1001]: 2026/09/23 04:43:21 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.ls worker1 # [ 59.683193] nix-grpc-daemon[906]: time=2026-09-23T04:43:21.202Z level=INFO msg="Upload complete. (289ms)" lb # [ 59.651781] niks3-server[1001]: 2026/09/23 04:43:21 INFO Registered completed upload object_key=log/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker1 # [ 59.692439] nix-grpc-daemon[770]: ts=2026-09-23T04:43:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=1 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv assign_id=6 outputs=1 lb # [ 59.662149] niks3-server[1001]: 2026/09/23 04:43:21 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete lb # [ 59.666896] niks3-server[1001]: 2026/09/23 04:43:21 INFO Completed upload id=12 lb # [ 59.670602] niks3-server[1001]: 2026/09/23 04:43:21 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.narinfo worker1 # [ 59.706114] nix-daemon[855]: reaped child process 956, status = succeeded worker1 # [ 59.709220] nix-grpc-daemon[770]: ts=2026-09-23T04:43:21Z level=debug event=want client=4 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv cached=0 worker1 # [ 59.714349] nix-grpc-daemon[770]: ts=2026-09-23T04:43:21Z level=debug event=assigned drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 59.762403] nix-grpc-daemon[770]: ts=2026-09-23T04:43:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 59.774266] nix-grpc-daemon[770]: ts=2026-09-23T04:43:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 59.784957] nix-grpc-daemon[906]: time=2026-09-23T04:43:21.304Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' to 'lb:50051'... worker1 # [ 59.794540] nix-grpc-daemon[770]: ts=2026-09-23T04:43:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 59.804752] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 59.931055] nix-daemon[855]: accepted connection from pid 983, user nix-grpc-daemon (trusted) worker1 # [ 59.955360] nix-daemon[855]: reaped child process 986, status = succeeded lb # [ 59.920422] niks3-server[1001]: 2026/09/23 04:43:21 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 59.962073] nix-grpc-daemon[906]: time=2026-09-23T04:43:21.481Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 59.963665] nix-grpc-daemon[906]: time=2026-09-23T04:43:21.481Z level=INFO msg="Uploading knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv (752B)" lb # [ 59.951360] niks3-server[1001]: 2026/09/23 04:43:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign worker1 # [ 59.990834] nix-grpc-daemon[906]: time=2026-09-23T04:43:21.510Z level=INFO msg="Uploading 1 narinfos" lb # [ 59.965736] niks3-server[1001]: 2026/09/23 04:43:21 INFO Signed narinfos id=13 count=1 lb # [ 59.973881] niks3-server[1001]: 2026/09/23 04:43:21 INFO Registered completed upload object_key=nar/0lpagyrngav67x80d1hq0njjlkmqdkvqawraz8dsp36ag50v6iwf.nar.zst lb # [ 59.988534] niks3-server[1001]: 2026/09/23 04:43:21 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.ls lb # [ 59.991626] niks3-server[1001]: 2026/09/23 04:43:21 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete worker1 # [ 60.029837] nix-grpc-daemon[906]: time=2026-09-23T04:43:21.549Z level=INFO msg="Upload complete. (214ms)" lb # [ 59.998783] niks3-server[1001]: 2026/09/23 04:43:21 INFO Completed upload id=13 lb # [ 60.001759] niks3-server[1001]: 2026/09/23 04:43:21 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.narinfo worker1 # [ 60.036144] nix-grpc-daemon[770]: ts=2026-09-23T04:43:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 60.045977] nix-grpc-daemon[770]: ts=2026-09-23T04:43:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 60.049924] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 60.053743] nix-daemon[855]: reaped child process 981, status = succeeded worker1 # [ 60.062233] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) client # building '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' on lb:50051... client # farm-top> node-a: building knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv lb # [ 60.178384] envoy[891]: conn peer=[2001:db8:1::1]:60118 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9562 tx=4361 ms=5915 lb # [ 60.183503] envoy[891]: conn peer=[2001:db8:1::1]:60100 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=480 ms=82 lb # [ 60.189255] envoy[891]: conn peer=[2001:db8:1::1]:60106 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4185 tx=1897 ms=2212 client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-sig worker1 # [ 61.152610] nix-daemon[855]: reaped child process 990, status = succeeded worker1 # [ 61.249069] nix-daemon[855]: accepted connection from pid 1006, user nix-grpc-daemon (trusted) worker1 # [ 61.272632] nix-daemon[855]: reaped child process 1009, status = succeeded lb # [ 61.245472] niks3-server[1001]: 2026/09/23 04:43:22 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 61.291472] nix-grpc-daemon[906]: time=2026-09-23T04:43:22.808Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 61.293526] nix-grpc-daemon[906]: time=2026-09-23T04:43:22.808Z level=INFO msg="Uploading 4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top (232B)" lb # [ 61.288535] niks3-server[1001]: 2026/09/23 04:43:22 INFO Registered completed upload object_key=nar/07ar364rwqqniwi2nkdpqlgv7ws7702n5lqz703ackfmssrh40v8.nar.zst lb # [ 61.301211] niks3-server[1001]: 2026/09/23 04:43:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign worker1 # [ 61.339638] nix-grpc-daemon[906]: time=2026-09-23T04:43:22.858Z level=INFO msg="Uploading 1 narinfos" lb # [ 61.311166] niks3-server[1001]: 2026/09/23 04:43:22 INFO Registered completed upload object_key=log/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv lb # [ 61.321600] niks3-server[1001]: 2026/09/23 04:43:22 INFO Signed narinfos id=14 count=1 lb # [ 61.323690] niks3-server[1001]: 2026/09/23 04:43:22 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete worker1 # [ 61.364040] nix-grpc-daemon[906]: time=2026-09-23T04:43:22.883Z level=INFO msg="Upload complete. (291ms)" lb # [ 61.332217] niks3-server[1001]: 2026/09/23 04:43:22 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.ls lb # [ 61.334941] niks3-server[1001]: 2026/09/23 04:43:22 INFO Completed upload id=14 lb # [ 61.338473] niks3-server[1001]: 2026/09/23 04:43:22 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.narinfo worker1 # [ 61.370976] nix-grpc-daemon[770]: ts=2026-09-23T04:43:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=1 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv assign_id=7 outputs=1 worker1 # [ 61.383138] nix-daemon[855]: reaped child process 988, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2, in 5.89 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath, in 0.19 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker1 # [ 61.752421] nix-grpc-daemon[770]: ts=2026-09-23T04:43:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:38428 worker1 # [ 61.794359] nix-grpc-daemon[770]: ts=2026-09-23T04:43:23Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:38428 client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.25 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker2 # [ 62.060711] nix-grpc-daemon[773]: ts=2026-09-23T04:43:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:43178 client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked): client # /nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a client # /nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b client # /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker2 # [ 62.107934] nix-grpc-daemon[773]: ts=2026-09-23T04:43:23Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:43178 worker2 # [ 62.117035] nix-grpc-daemon[773]: copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'http://lb:5751'... worker2 # [ 62.134056] nix-grpc-daemon[773]: copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'http://lb:5751'... worker2 # [ 62.148653] nix-grpc-daemon[773]: copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.32 seconds) client: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top client # finding garbage collector roots... client # removing stale temporary roots file "/nix/var/nix/temproots/849" client # removing stale temporary roots file "/nix/var/nix/temproots/906" client # removing stale temporary roots file "/nix/var/nix/temproots/886" client # removing stale temporary roots file "/nix/var/nix/temproots/898" client # removing stale temporary roots file "/nix/var/nix/temproots/927" client # removing stale temporary roots file "/nix/var/nix/temproots/919" client # removing stale temporary roots file "/nix/var/nix/temproots/865" client # removing stale temporary roots file "/nix/var/nix/temproots/942" client # deleting specified paths... client # deleting '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' client # deleting unused links... client # note: hard linking is currently saving 0.0 KiB worker2 # [ 62.543526] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 62.550098] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 62.558824] nix-grpc-daemon[856]: time=2026-09-23T04:43:24.076Z level=INFO msg="All 1 paths already cached" worker2 # [ 62.564372] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 62.570817] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:32768 client # copying 3 paths... worker2 # [ 62.581219] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:32768 client # copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'grpc://lb:50051'... worker2 # [ 62.602854] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 62.610474] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_out=232 worker1 # [ 62.605238] nix-grpc-daemon[770]: ts=2026-09-23T04:43:24Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 62.613176] nix-grpc-daemon[770]: ts=2026-09-23T04:43:24Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=0 paths=1 nar_bytes_out=176 worker2 # [ 62.628489] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 62.635519] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_out=128 client # copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'grpc://lb:50051'... client # copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'grpc://lb:50051'... client: (finished: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.54 seconds) (finished: subtest: outputs read back through any worker carry the cache signature, in 7.18 seconds) subtest: two clients wanting the same drv build it once client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup client # [ 62.756751] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup1.service; invocation ID: ad6f64b680fc47738f58a1c368887b80 client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.06 seconds) client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup worker2 # [ 62.971429] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:32768 client # [ 62.958294] nix[995]: don't know how to build these paths: client # [ 62.959435] nix[995]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker1 # [ 62.978171] nix-grpc-daemon[770]: ts=2026-09-23T04:43:24Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker1 # [ 62.983536] nix-grpc-daemon[770]: ts=2026-09-23T04:43:24Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 63.039306] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 63.056096] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 client # [ 63.052176] nix[995]: copying 1 paths to 'lb:50051'... worker2 # [ 63.073122] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 client # [ 63.054534] nix[995]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'... worker2 # [ 63.089157] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 63.251400] nix-daemon[843]: accepted connection from pid 1006, user nix-grpc-daemon (trusted) worker2 # [ 63.272580] nix-daemon[843]: reaped child process 1009, status = succeeded lb # [ 63.234904] niks3-server[1001]: 2026/09/23 04:43:24 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 63.289488] nix-grpc-daemon[856]: time=2026-09-23T04:43:24.807Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 63.291094] nix-grpc-daemon[856]: time=2026-09-23T04:43:24.807Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)" lb # [ 63.269913] niks3-server[1001]: 2026/09/23 04:43:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign worker2 # [ 63.328121] nix-grpc-daemon[856]: time=2026-09-23T04:43:24.846Z level=INFO msg="Uploading 1 narinfos" lb # [ 63.288410] niks3-server[1001]: 2026/09/23 04:43:24 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst lb # [ 63.294388] niks3-server[1001]: 2026/09/23 04:43:24 INFO Signed narinfos id=15 count=1 lb # [ 63.303227] niks3-server[1001]: 2026/09/23 04:43:24 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls lb # [ 63.313454] niks3-server[1001]: 2026/09/23 04:43:24 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete worker2 # [ 63.363308] nix-grpc-daemon[856]: time=2026-09-23T04:43:24.881Z level=INFO msg="Upload complete. (256ms)" lb # [ 63.321130] niks3-server[1001]: 2026/09/23 04:43:24 INFO Completed upload id=15 lb # [ 63.324142] niks3-server[1001]: 2026/09/23 04:43:24 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo worker2 # [ 63.377297] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_in=432 worker2 # [ 63.391892] nix-grpc-daemon[773]: ts=2026-09-23T04:43:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 63.400414] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 63.406265] nix-daemon[843]: reaped child process 1004, status = succeeded worker2 # [ 63.417259] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) client # [ 63.397085] nix[995]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... client # [ 63.852873] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup2.service; invocation ID: 54f9ab5bbf434548a16b761b7de44b4e client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.12 seconds) worker1 # [ 64.098428] nix-grpc-daemon[770]: ts=2026-09-23T04:43:25Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59106 client # [ 64.099441] nix[1011]: don't know how to build these paths: client # [ 64.100692] nix[1011]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker1 # [ 64.115144] nix-grpc-daemon[770]: ts=2026-09-23T04:43:25Z level=debug event=want client=6 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker2 # [ 64.134263] nix-grpc-daemon[773]: ts=2026-09-23T04:43:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 client # [ 64.122404] nix[1011]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... client: must succeed: sleep 2 client: (finished: must succeed: sleep 2, in 2.05 seconds) client: waiting for success: ! systemctl is-active dedup1 dedup2 worker2 # [ 67.446656] nix-daemon[843]: reaped child process 1013, status = succeeded worker2 # [ 67.546985] nix-daemon[843]: accepted connection from pid 1043, user nix-grpc-daemon (trusted) worker2 # [ 67.570292] nix-daemon[843]: reaped child process 1046, status = succeeded lb # [ 67.525175] niks3-server[1001]: 2026/09/23 04:43:29 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 67.588906] nix-grpc-daemon[856]: time=2026-09-23T04:43:29.107Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 67.594021] nix-grpc-daemon[856]: time=2026-09-23T04:43:29.109Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)" lb # [ 67.579803] niks3-server[1001]: 2026/09/23 04:43:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign worker2 # [ 67.633348] nix-grpc-daemon[856]: time=2026-09-23T04:43:29.151Z level=INFO msg="Uploading 1 narinfos" lb # [ 67.595493] niks3-server[1001]: 2026/09/23 04:43:29 INFO Signed narinfos id=16 count=1 lb # [ 67.603564] niks3-server[1001]: 2026/09/23 04:43:29 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv lb # [ 67.612230] niks3-server[1001]: 2026/09/23 04:43:29 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls lb # [ 67.615197] niks3-server[1001]: 2026/09/23 04:43:29 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst lb # [ 67.628240] niks3-server[1001]: 2026/09/23 04:43:29 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete lb # [ 67.634685] niks3-server[1001]: 2026/09/23 04:43:29 INFO Completed upload id=16 lb # [ 67.638416] niks3-server[1001]: 2026/09/23 04:43:29 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo worker2 # [ 67.679421] nix-grpc-daemon[856]: time=2026-09-23T04:43:29.197Z level=INFO msg="Upload complete. (247ms)" worker2 # [ 67.688954] nix-grpc-daemon[773]: ts=2026-09-23T04:43:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker2 # [ 67.700238] nix-grpc-daemon[773]: ts=2026-09-23T04:43:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker2 # [ 67.707023] nix-daemon[843]: reaped child process 1011, status = succeeded client # [ 67.704902] systemd[1]: dedup2.service: Deactivated successfully. client # [ 67.709871] systemd[1]: dedup2.service: Consumed 208ms CPU time over 3.856s wall clock time, 35.2M memory peak, 5.9K incoming IP traffic, 5.6K outgoing IP traffic. client # [ 67.747104] systemd[1]: dedup1.service: Deactivated successfully. client # [ 67.750255] systemd[1]: dedup1.service: Consumed 176ms CPU time over 4.993s wall clock time, 39.5M memory peak, 7.2K incoming IP traffic, 8.5K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.13 seconds) client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; } client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.06 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: two clients wanting the same drv build it once, in 6.13 seconds) subtest: --timeout and --max-silent-time reach the builder worker1 # [ 69.087490] nix-grpc-daemon[770]: ts=2026-09-23T04:43:30Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 69.105299] nix-grpc-daemon[770]: ts=2026-09-23T04:43:30Z level=debug event=want client=7 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv cached=0 worker1 # [ 69.112500] nix-grpc-daemon[770]: ts=2026-09-23T04:43:30Z level=debug event=assigned drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 69.121626] nix-grpc-daemon[770]: ts=2026-09-23T04:43:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 69.133698] nix-grpc-daemon[770]: ts=2026-09-23T04:43:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 69.151775] nix-grpc-daemon[770]: ts=2026-09-23T04:43:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 69.165977] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 69.323042] nix-daemon[855]: accepted connection from pid 1040, user nix-grpc-daemon (trusted) worker1 # [ 69.348293] nix-daemon[855]: reaped child process 1043, status = succeeded lb # [ 69.321720] niks3-server[1001]: 2026/09/23 04:43:30 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 69.364516] nix-grpc-daemon[906]: time=2026-09-23T04:43:30.883Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 69.366474] nix-grpc-daemon[906]: time=2026-09-23T04:43:30.883Z level=INFO msg="Uploading sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv (424B)" lb # [ 69.360248] niks3-server[1001]: 2026/09/23 04:43:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign worker1 # [ 69.401096] nix-grpc-daemon[906]: time=2026-09-23T04:43:30.920Z level=INFO msg="Uploading 1 narinfos" lb # [ 69.372687] niks3-server[1001]: 2026/09/23 04:43:30 INFO Signed narinfos id=17 count=1 lb # [ 69.380114] niks3-server[1001]: 2026/09/23 04:43:30 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.ls lb # [ 69.388580] niks3-server[1001]: 2026/09/23 04:43:30 INFO Registered completed upload object_key=nar/1m12xmj6yzmb9y2j8ly0yvh5fyzy1nqk7ajhqwr4wn3b5h0bsch5.nar.zst worker1 # [ 69.435065] nix-grpc-daemon[906]: time=2026-09-23T04:43:30.954Z level=INFO msg="Upload complete. (253ms)" lb # [ 69.403269] niks3-server[1001]: 2026/09/23 04:43:30 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete lb # [ 69.411507] niks3-server[1001]: 2026/09/23 04:43:30 INFO Completed upload id=17 lb # [ 69.416378] niks3-server[1001]: 2026/09/23 04:43:30 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.narinfo worker1 # [ 69.448973] nix-grpc-daemon[770]: ts=2026-09-23T04:43:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=0 paths=1 nar_bytes_in=424 worker1 # [ 69.463722] nix-grpc-daemon[770]: ts=2026-09-23T04:43:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 69.469857] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 69.474537] nix-daemon[855]: reaped child process 1038, status = succeeded worker1 # [ 69.487652] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) lb # [ 70.181363] envoy[891]: conn peer=[2001:db8:1::1]:60108 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9584 tx=4414 ms=5650 lb # [ 70.186405] envoy[891]: conn peer=[2001:db8:1::1]:46540 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1051 tx=2887 ms=165 lb # [ 70.192161] envoy[891]: conn peer=[2001:db8:1::1]:46572 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=330 ms=81 lb # [ 70.196901] envoy[891]: conn peer=[2001:db8:1::1]:46566 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=408 tx=376 ms=94 lb # [ 70.201532] envoy[891]: conn peer=[2001:db8:1::1]:46550 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=409 ms=107 lb # [ 70.206973] envoy[891]: conn peer=[2001:db8:1::1]:46588 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1198 ms=3603 lb # [ 70.212183] envoy[891]: conn peer=[2001:db8:1::1]:46576 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3018 tx=1541 ms=4779 worker1 # [ 75.305703] nix-grpc-daemon[770]: ts=2026-09-23T04:43:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=5 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv assign_id=9 outputs=0 worker1 # [ 75.347652] nix-daemon[855]: reaped child process 1045, status = succeeded worker1 # [ 75.350795] nix-daemon[855]: reaped child process 1047, status = succeeded worker1 # [ 75.550309] nix-grpc-daemon[770]: ts=2026-09-23T04:43:37Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 75.566190] nix-grpc-daemon[770]: ts=2026-09-23T04:43:37Z level=debug event=want client=8 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv cached=0 worker1 # [ 75.573036] nix-grpc-daemon[770]: ts=2026-09-23T04:43:37Z level=debug event=assigned drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 75.589675] nix-grpc-daemon[773]: ts=2026-09-23T04:43:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 75.608321] nix-grpc-daemon[773]: ts=2026-09-23T04:43:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 75.623140] nix-grpc-daemon[773]: ts=2026-09-23T04:43:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 75.638467] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 75.800256] nix-daemon[843]: accepted connection from pid 1059, user nix-grpc-daemon (trusted) worker2 # [ 75.821044] nix-daemon[843]: reaped child process 1062, status = succeeded lb # [ 75.783815] niks3-server[1001]: 2026/09/23 04:43:37 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 75.840744] nix-grpc-daemon[856]: time=2026-09-23T04:43:37.358Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 75.842418] nix-grpc-daemon[856]: time=2026-09-23T04:43:37.358Z level=INFO msg="Uploading pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv (424B)" lb # [ 75.823664] niks3-server[1001]: 2026/09/23 04:43:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign worker2 # [ 75.878420] nix-grpc-daemon[856]: time=2026-09-23T04:43:37.396Z level=INFO msg="Uploading 1 narinfos" lb # [ 75.836288] niks3-server[1001]: 2026/09/23 04:43:37 INFO Signed narinfos id=18 count=1 lb # [ 75.846226] niks3-server[1001]: 2026/09/23 04:43:37 INFO Registered completed upload object_key=nar/14f5yrnllfav3svmw4v4svw15r4zlf5yv1p972jy14mmq6gdw943.nar.zst lb # [ 75.856308] niks3-server[1001]: 2026/09/23 04:43:37 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.ls lb # [ 75.866722] niks3-server[1001]: 2026/09/23 04:43:37 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete lb # [ 75.873697] niks3-server[1001]: 2026/09/23 04:43:37 INFO Completed upload id=18 worker2 # [ 75.917179] nix-grpc-daemon[856]: time=2026-09-23T04:43:37.435Z level=INFO msg="Upload complete. (261ms)" lb # [ 75.877182] niks3-server[1001]: 2026/09/23 04:43:37 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.narinfo worker2 # [ 75.928369] nix-grpc-daemon[773]: ts=2026-09-23T04:43:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 75.948118] nix-grpc-daemon[773]: ts=2026-09-23T04:43:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 75.955905] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 75.962187] nix-daemon[843]: reaped child process 1057, status = succeeded worker2 # [ 75.973695] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) lb # [ 80.184559] envoy[891]: conn peer=[2001:db8:1::1]:46592 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3027 tx=952 ms=6249 worker2 # [ 81.791640] nix-grpc-daemon[773]: ts=2026-09-23T04:43:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=5 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv assign_id=10 outputs=0 worker2 # [ 81.830102] nix-daemon[843]: reaped child process 1064, status = succeeded worker2 # [ 81.834067] nix-daemon[843]: reaped child process 1066, status = succeeded (finished: subtest: --timeout and --max-silent-time reach the builder, in 13.16 seconds) subtest: the first client leaving does not fail the second client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share client # [ 82.069547] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share1.service; invocation ID: f9331f6e64c044a5b437aa847330800d client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 0.09 seconds) client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share worker2 # [ 82.298052] nix-grpc-daemon[773]: ts=2026-09-23T04:43:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:32768 client # [ 82.285836] nix[1085]: don't know how to build these paths: client # [ 82.286810] nix[1085]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker1 # [ 82.305426] nix-grpc-daemon[770]: ts=2026-09-23T04:43:43Z level=debug event=want client=9 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 worker1 # [ 82.311952] nix-grpc-daemon[770]: ts=2026-09-23T04:43:43Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 82.322329] nix-grpc-daemon[770]: ts=2026-09-23T04:43:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 82.338029] nix-grpc-daemon[770]: ts=2026-09-23T04:43:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59106 client # [ 82.343477] nix[1085]: copying 1 paths to 'lb:50051'... client # [ 82.347776] nix[1085]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'... worker1 # [ 82.355743] nix-grpc-daemon[770]: ts=2026-09-23T04:43:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 82.371740] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 82.533346] nix-daemon[855]: accepted connection from pid 1074, user nix-grpc-daemon (trusted) worker1 # [ 82.551644] nix-daemon[855]: reaped child process 1077, status = succeeded lb # [ 82.530259] niks3-server[1001]: 2026/09/23 04:43:44 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 82.571438] nix-grpc-daemon[906]: time=2026-09-23T04:43:44.090Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 82.573389] nix-grpc-daemon[906]: time=2026-09-23T04:43:44.090Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)" lb # [ 82.563229] niks3-server[1001]: 2026/09/23 04:43:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign worker1 # [ 82.604285] nix-grpc-daemon[906]: time=2026-09-23T04:43:44.123Z level=INFO msg="Uploading 1 narinfos" lb # [ 82.575374] niks3-server[1001]: 2026/09/23 04:43:44 INFO Signed narinfos id=19 count=1 lb # [ 82.585754] niks3-server[1001]: 2026/09/23 04:43:44 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls lb # [ 82.593959] niks3-server[1001]: 2026/09/23 04:43:44 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst lb # [ 82.607419] niks3-server[1001]: 2026/09/23 04:43:44 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete lb # [ 82.612485] niks3-server[1001]: 2026/09/23 04:43:44 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo worker1 # [ 82.644813] nix-grpc-daemon[906]: time=2026-09-23T04:43:44.164Z level=INFO msg="Upload complete. (255ms)" lb # [ 82.615786] niks3-server[1001]: 2026/09/23 04:43:44 INFO Completed upload id=19 worker1 # [ 82.656484] nix-grpc-daemon[770]: ts=2026-09-23T04:43:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=0 paths=1 nar_bytes_in=432 worker1 # [ 82.668694] nix-grpc-daemon[770]: ts=2026-09-23T04:43:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 82.680612] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 82.686453] nix-daemon[855]: reaped child process 1072, status = succeeded client # [ 82.688952] nix[1085]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... worker1 # [ 82.699430] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) client # [ 83.177600] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share2.service; invocation ID: 413e0c2530d5482abee3d20c4723c143 client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 1.12 seconds) worker2 # [ 83.415095] nix-grpc-daemon[773]: ts=2026-09-23T04:43:44Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:32768 client # [ 83.404381] nix[1101]: don't know how to build these paths: client # [ 83.405469] nix[1101]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker1 # [ 83.422002] nix-grpc-daemon[770]: ts=2026-09-23T04:43:44Z level=debug event=want client=10 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 client # [ 83.423553] nix[1101]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... worker1 # [ 83.430138] nix-grpc-daemon[770]: ts=2026-09-23T04:43:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 client: must succeed: sleep 2; systemctl kill -s INT share1 client # [ 86.565476] systemd[1]: share1.service: Sent signal SIGINT to main process 1085 (nix) on client request. client # [ 86.576886] nix[1085]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client # [ 86.584710] nix[1085]: [mutex.cc : 451] RAW: Unlock 0xfbf7089fc788 @ 0xfbfb0d0359a4 0xfbfb0e10d40c 0xfbfb2d5957bc 0xfbfb2d597414 0xfbfb2d58e1d4 0xfbfb2c5d4f7c 0xfbfb2c362d80 client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.09 seconds) client: waiting for success: ! systemctl is-active share1 client # [ 86.616851] systemd-coredump[1116]: Process 1085 (nix) of user 0 terminated abnormally with signal 11/SEGV, processing... client # [ 86.645683] systemd[1]: Created slice Slice /system/systemd-coredump. client # [ 86.653985] systemd[1]: Started Process Core Dump (PID 1116/UID 0). client # [ 89.158303] systemd-coredump[1121]: Cannot store coredump of 1085 (nix): No space left on device client # [ 89.288918] systemd-coredump[1121]: Process 1085 (nix) of user 0 terminated abnormally without generating a coredump. client # [ 89.309073] systemd[1]: systemd-coredump@0-1-1116_1117-0.service: Deactivated successfully. client # [ 89.310668] systemd[1]: systemd-coredump@0-1-1116_1117-0.service: Consumed 2.106s CPU time over 2.647s wall clock time, 968.8M memory peak, 926.5M written to disk. client # [ 89.319667] systemd[1]: share1.service: Main process exited, code=dumped, status=11/SEGV client # [ 89.323076] systemd[1]: share1.service: Failed with result 'core-dump'. client # [ 89.324513] systemd[1]: share1.service: Consumed 398ms CPU time over 7.253s wall clock time, 41.5M memory peak, 4K written to disk, 6.2K incoming IP traffic, 8.2K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share1, in 3.29 seconds) worker1 # [ 90.132610] nix-daemon[855]: reaped child process 1081, status = succeeded client: waiting for success: ! systemctl is-active share2 lb # [ 90.187818] envoy[891]: conn peer=[2001:db8:1::1]:39716 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2989 tx=977 ms=6293 lb # [ 90.193117] envoy[891]: conn peer=[2001:db8:1::1]:38418 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3029 tx=788 ms=7035 worker1 # [ 90.233800] nix-daemon[855]: accepted connection from pid 1117, user nix-grpc-daemon (trusted) worker1 # [ 90.257995] nix-daemon[855]: reaped child process 1120, status = succeeded lb # [ 90.230457] niks3-server[1001]: 2026/09/23 04:43:51 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 90.275577] nix-grpc-daemon[906]: time=2026-09-23T04:43:51.792Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 90.277275] nix-grpc-daemon[906]: time=2026-09-23T04:43:51.792Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)" lb # [ 90.285309] niks3-server[1001]: 2026/09/23 04:43:51 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv lb # [ 90.296758] niks3-server[1001]: 2026/09/23 04:43:51 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst worker1 # [ 90.339094] nix-grpc-daemon[906]: time=2026-09-23T04:43:51.858Z level=INFO msg="Uploading 1 narinfos" lb # [ 90.306476] niks3-server[1001]: 2026/09/23 04:43:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign lb # [ 90.314435] niks3-server[1001]: 2026/09/23 04:43:51 INFO Signed narinfos id=20 count=1 lb # [ 90.318762] niks3-server[1001]: 2026/09/23 04:43:51 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls lb # [ 90.327890] niks3-server[1001]: 2026/09/23 04:43:51 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete worker1 # [ 90.365934] nix-grpc-daemon[906]: time=2026-09-23T04:43:51.885Z level=INFO msg="Upload complete. (249ms)" lb # [ 90.334175] niks3-server[1001]: 2026/09/23 04:43:51 INFO Completed upload id=20 lb # [ 90.339124] niks3-server[1001]: 2026/09/23 04:43:51 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo worker1 # [ 90.381410] nix-grpc-daemon[770]: ts=2026-09-23T04:43:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=6 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 worker1 # [ 90.399508] nix-grpc-daemon[770]: ts=2026-09-23T04:43:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=7 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 worker1 # [ 90.405551] nix-daemon[855]: reaped child process 1079, status = succeeded client # [ 90.461038] systemd[1]: share2.service: Deactivated successfully. client # [ 90.466239] systemd[1]: share2.service: Consumed 200ms CPU time over 7.289s wall clock time, 33.3M memory peak, 4K written to disk, 6.1K incoming IP traffic, 5.7K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share2, in 1.18 seconds) client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; } client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.07 seconds) (finished: subtest: the first client leaving does not fail the second, in 9.40 seconds) subtest: build hook: nix-daemon with builders = grpc://lb client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log client # [ 91.588446] systemd[1]: Started Nix Daemon. client # [ 91.748881] nix-daemon[1167]: accepted connection from pid 1164, user root (trusted) worker1 # [ 91.967116] nix-grpc-daemon[770]: ts=2026-09-23T04:43:53Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:59106 worker2 # [ 91.988424] nix-grpc-daemon[773]: ts=2026-09-23T04:43:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker1 # [ 91.994010] nix-grpc-daemon[770]: ts=2026-09-23T04:43:53Z level=debug event=want client=11 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=0 worker1 # [ 92.001823] nix-grpc-daemon[770]: ts=2026-09-23T04:43:53Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 92.018998] nix-grpc-daemon[773]: ts=2026-09-23T04:43:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 92.040902] nix-grpc-daemon[773]: ts=2026-09-23T04:43:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 92.062845] nix-grpc-daemon[773]: ts=2026-09-23T04:43:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 92.082164] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 92.320209] nix-daemon[843]: accepted connection from pid 1105, user nix-grpc-daemon (trusted) worker2 # [ 92.342960] nix-daemon[843]: reaped child process 1108, status = succeeded lb # [ 92.311221] niks3-server[1001]: 2026/09/23 04:43:53 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 92.366036] nix-grpc-daemon[856]: time=2026-09-23T04:43:53.883Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 92.367653] nix-grpc-daemon[856]: time=2026-09-23T04:43:53.883Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)" worker2 # [ 92.420111] nix-grpc-daemon[856]: time=2026-09-23T04:43:53.937Z level=INFO msg="Uploading 1 narinfos" lb # [ 92.358938] niks3-server[1001]: 2026/09/23 04:43:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign lb # [ 92.379993] niks3-server[1001]: 2026/09/23 04:43:53 INFO Signed narinfos id=21 count=1 lb # [ 92.388807] niks3-server[1001]: 2026/09/23 04:43:53 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst lb # [ 92.398901] niks3-server[1001]: 2026/09/23 04:43:53 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls lb # [ 92.413979] niks3-server[1001]: 2026/09/23 04:43:53 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete worker2 # [ 92.469020] nix-grpc-daemon[856]: time=2026-09-23T04:43:53.987Z level=INFO msg="Upload complete. (361ms)" lb # [ 92.426800] niks3-server[1001]: 2026/09/23 04:43:53 INFO Completed upload id=21 lb # [ 92.429708] niks3-server[1001]: 2026/09/23 04:43:53 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo worker2 # [ 92.485109] nix-grpc-daemon[773]: ts=2026-09-23T04:43:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 92.496605] nix-grpc-daemon[773]: ts=2026-09-23T04:43:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 92.519128] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 92.528264] nix-daemon[843]: reaped child process 1103, status = succeeded worker2 # [ 92.546299] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 93.801370] nix-daemon[843]: reaped child process 1112, status = succeeded worker2 # [ 93.906707] nix-daemon[843]: accepted connection from pid 1128, user nix-grpc-daemon (trusted) worker2 # [ 93.930387] nix-daemon[843]: reaped child process 1131, status = succeeded lb # [ 93.885269] niks3-server[1001]: 2026/09/23 04:43:55 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 93.942429] nix-grpc-daemon[856]: time=2026-09-23T04:43:55.458Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 93.944070] nix-grpc-daemon[856]: time=2026-09-23T04:43:55.459Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)" lb # [ 93.927008] niks3-server[1001]: 2026/09/23 04:43:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign worker2 # [ 93.981457] nix-grpc-daemon[856]: time=2026-09-23T04:43:55.499Z level=INFO msg="Uploading 1 narinfos" lb # [ 93.941251] niks3-server[1001]: 2026/09/23 04:43:55 INFO Signed narinfos id=22 count=1 lb # [ 93.949486] niks3-server[1001]: 2026/09/23 04:43:55 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls worker2 # [ 94.004990] nix-grpc-daemon[856]: time=2026-09-23T04:43:55.523Z level=INFO msg="Upload complete. (242ms)" lb # [ 93.957816] niks3-server[1001]: 2026/09/23 04:43:55 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv lb # [ 93.969109] niks3-server[1001]: 2026/09/23 04:43:55 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst worker2 # [ 94.014160] nix-grpc-daemon[773]: ts=2026-09-23T04:43:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=12 outputs=1 lb # [ 93.975488] niks3-server[1001]: 2026/09/23 04:43:55 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete lb # [ 93.979895] niks3-server[1001]: 2026/09/23 04:43:55 INFO Completed upload id=22 lb # [ 93.983518] niks3-server[1001]: 2026/09/23 04:43:55 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo worker2 # [ 94.031893] nix-grpc-daemon[773]: ts=2026-09-23T04:43:55Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 94.036080] nix-daemon[843]: reaped child process 1110, status = succeeded worker2 # [ 94.049975] nix-grpc-daemon[773]: ts=2026-09-23T04:43:55Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 94.055486] nix-grpc-daemon[773]: ts=2026-09-23T04:43:55Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_out=128 worker1 # [ 94.337720] nix-grpc-daemon[770]: ts=2026-09-23T04:43:55Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:59106 worker2 # [ 94.352639] nix-grpc-daemon[773]: ts=2026-09-23T04:43:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 94.417944] nix-grpc-daemon[856]: time=2026-09-23T04:43:55.936Z level=INFO msg="All 1 paths already cached" worker1 # [ 94.428252] nix-grpc-daemon[770]: ts=2026-09-23T04:43:55Z level=debug event=want client=12 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=0 worker1 # [ 94.434681] nix-grpc-daemon[770]: ts=2026-09-23T04:43:55Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 94.493691] nix-grpc-daemon[773]: ts=2026-09-23T04:43:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 94.509401] nix-grpc-daemon[773]: ts=2026-09-23T04:43:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 94.527195] nix-grpc-daemon[856]: time=2026-09-23T04:43:56.045Z level=INFO msg="All 1 paths already cached" worker2 # [ 94.537353] nix-grpc-daemon[773]: ts=2026-09-23T04:43:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 94.551756] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 94.719693] nix-daemon[843]: accepted connection from pid 1135, user nix-grpc-daemon (trusted) worker2 # [ 94.739011] nix-daemon[843]: reaped child process 1138, status = succeeded lb # [ 94.698199] niks3-server[1001]: 2026/09/23 04:43:56 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 94.752887] nix-grpc-daemon[856]: time=2026-09-23T04:43:56.271Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 94.754570] nix-grpc-daemon[856]: time=2026-09-23T04:43:56.271Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)" lb # [ 94.728994] niks3-server[1001]: 2026/09/23 04:43:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign worker2 # [ 94.782148] nix-grpc-daemon[856]: time=2026-09-23T04:43:56.300Z level=INFO msg="Uploading 1 narinfos" lb # [ 94.741301] niks3-server[1001]: 2026/09/23 04:43:56 INFO Signed narinfos id=23 count=1 lb # [ 94.751350] niks3-server[1001]: 2026/09/23 04:43:56 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls lb # [ 94.761542] niks3-server[1001]: 2026/09/23 04:43:56 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst worker2 # [ 94.817042] nix-grpc-daemon[856]: time=2026-09-23T04:43:56.335Z level=INFO msg="Upload complete. (247ms)" lb # [ 94.775633] niks3-server[1001]: 2026/09/23 04:43:56 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete lb # [ 94.785233] niks3-server[1001]: 2026/09/23 04:43:56 INFO Completed upload id=23 worker2 # [ 94.828533] nix-grpc-daemon[773]: ts=2026-09-23T04:43:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 94.788105] niks3-server[1001]: 2026/09/23 04:43:56 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo worker2 # [ 94.844508] nix-grpc-daemon[773]: ts=2026-09-23T04:43:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 94.852444] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 94.857751] nix-daemon[843]: reaped child process 1133, status = succeeded worker2 # [ 94.869264] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 95.497016] nix-daemon[843]: reaped child process 1142, status = succeeded worker2 # [ 95.609556] nix-daemon[843]: accepted connection from pid 1158, user nix-grpc-daemon (trusted) worker2 # [ 95.632903] nix-daemon[843]: reaped child process 1161, status = succeeded lb # [ 95.594150] niks3-server[1001]: 2026/09/23 04:43:57 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 95.648426] nix-grpc-daemon[856]: time=2026-09-23T04:43:57.165Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 95.649895] nix-grpc-daemon[856]: time=2026-09-23T04:43:57.165Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)" lb # [ 95.638460] niks3-server[1001]: 2026/09/23 04:43:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign worker2 # [ 95.688346] nix-grpc-daemon[856]: time=2026-09-23T04:43:57.206Z level=INFO msg="Uploading 1 narinfos" lb # [ 95.651336] niks3-server[1001]: 2026/09/23 04:43:57 INFO Signed narinfos id=24 count=1 lb # [ 95.658913] niks3-server[1001]: 2026/09/23 04:43:57 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst worker2 # [ 95.715236] nix-grpc-daemon[856]: time=2026-09-23T04:43:57.233Z level=INFO msg="Upload complete. (234ms)" lb # [ 95.671290] niks3-server[1001]: 2026/09/23 04:43:57 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv lb # [ 95.678548] niks3-server[1001]: 2026/09/23 04:43:57 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls lb # [ 95.684331] niks3-server[1001]: 2026/09/23 04:43:57 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete lb # [ 95.690730] niks3-server[1001]: 2026/09/23 04:43:57 INFO Completed upload id=24 worker2 # [ 95.724967] nix-grpc-daemon[773]: ts=2026-09-23T04:43:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=13 outputs=1 lb # [ 95.693352] niks3-server[1001]: 2026/09/23 04:43:57 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo worker2 # [ 95.740974] nix-daemon[843]: reaped child process 1140, status = succeeded worker1 # [ 95.765712] nix-grpc-daemon[770]: ts=2026-09-23T04:43:57Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 95.786856] nix-grpc-daemon[770]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'... worker1 # [ 95.804924] nix-grpc-daemon[770]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'... worker2 # [ 95.857360] nix-grpc-daemon[773]: ts=2026-09-23T04:43:57Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 95.864627] nix-grpc-daemon[773]: ts=2026-09-23T04:43:57Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_out=176 worker1 # [ 96.102826] nix-grpc-daemon[770]: ts=2026-09-23T04:43:57Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:59106 worker2 # [ 96.121067] nix-grpc-daemon[773]: ts=2026-09-23T04:43:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 96.343680] nix-grpc-daemon[856]: time=2026-09-23T04:43:57.862Z level=INFO msg="All 2 paths already cached" worker1 # [ 96.356440] nix-grpc-daemon[770]: ts=2026-09-23T04:43:57Z level=debug event=want client=13 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=0 worker1 # [ 96.362953] nix-grpc-daemon[770]: ts=2026-09-23T04:43:57Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 96.379643] nix-grpc-daemon[773]: ts=2026-09-23T04:43:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 96.397706] nix-grpc-daemon[773]: ts=2026-09-23T04:43:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 96.417395] nix-grpc-daemon[856]: time=2026-09-23T04:43:57.935Z level=INFO msg="All 2 paths already cached" worker2 # [ 96.428922] nix-grpc-daemon[773]: ts=2026-09-23T04:43:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 96.443911] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 96.608411] nix-daemon[843]: accepted connection from pid 1165, user nix-grpc-daemon (trusted) worker2 # [ 96.629192] nix-daemon[843]: reaped child process 1168, status = succeeded lb # [ 96.589447] niks3-server[1001]: 2026/09/23 04:43:58 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 96.645478] nix-grpc-daemon[856]: time=2026-09-23T04:43:58.163Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 96.647167] nix-grpc-daemon[856]: time=2026-09-23T04:43:58.163Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)" lb # [ 96.622534] niks3-server[1001]: 2026/09/23 04:43:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign worker2 # [ 96.677922] nix-grpc-daemon[856]: time=2026-09-23T04:43:58.195Z level=INFO msg="Uploading 1 narinfos" lb # [ 96.642730] niks3-server[1001]: 2026/09/23 04:43:58 INFO Signed narinfos id=25 count=1 lb # [ 96.648964] niks3-server[1001]: 2026/09/23 04:43:58 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls lb # [ 96.659650] niks3-server[1001]: 2026/09/23 04:43:58 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst lb # [ 96.663919] niks3-server[1001]: 2026/09/23 04:43:58 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete lb # [ 96.669279] niks3-server[1001]: 2026/09/23 04:43:58 INFO Completed upload id=25 worker2 # [ 96.712637] nix-grpc-daemon[856]: time=2026-09-23T04:43:58.231Z level=INFO msg="Upload complete. (250ms)" lb # [ 96.674564] niks3-server[1001]: 2026/09/23 04:43:58 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo worker2 # [ 96.725413] nix-grpc-daemon[773]: ts=2026-09-23T04:43:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_in=752 worker2 # [ 96.740241] nix-grpc-daemon[773]: ts=2026-09-23T04:43:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 96.746514] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 96.751241] nix-daemon[843]: reaped child process 1163, status = succeeded worker2 # [ 96.763220] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 98.470879] nix-daemon[843]: reaped child process 1172, status = succeeded worker2 # [ 98.538676] nix-daemon[843]: accepted connection from pid 1188, user nix-grpc-daemon (trusted) worker2 # [ 98.560471] nix-daemon[843]: reaped child process 1191, status = succeeded lb # [ 98.521870] niks3-server[1001]: 2026/09/23 04:44:00 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 98.576427] nix-grpc-daemon[856]: time=2026-09-23T04:44:00.094Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 98.578135] nix-grpc-daemon[856]: time=2026-09-23T04:44:00.094Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)" worker2 # [ 98.623142] nix-grpc-daemon[856]: time=2026-09-23T04:44:00.141Z level=INFO msg="Uploading 1 narinfos" lb # [ 98.567102] niks3-server[1001]: 2026/09/23 04:44:00 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv lb # [ 98.582618] niks3-server[1001]: 2026/09/23 04:44:00 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst lb # [ 98.593128] niks3-server[1001]: 2026/09/23 04:44:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign lb # [ 98.603867] niks3-server[1001]: 2026/09/23 04:44:00 INFO Signed narinfos id=26 count=1 worker2 # [ 98.648456] nix-grpc-daemon[856]: time=2026-09-23T04:44:00.167Z level=INFO msg="Upload complete. (272ms)" lb # [ 98.609987] niks3-server[1001]: 2026/09/23 04:44:00 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls lb # [ 98.616176] niks3-server[1001]: 2026/09/23 04:44:00 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete worker2 # [ 98.661115] nix-grpc-daemon[773]: ts=2026-09-23T04:44:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=14 outputs=1 lb # [ 98.621454] niks3-server[1001]: 2026/09/23 04:44:00 INFO Completed upload id=26 lb # [ 98.624979] niks3-server[1001]: 2026/09/23 04:44:00 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo worker2 # [ 98.679250] nix-daemon[843]: reaped child process 1170, status = succeeded worker1 # [ 98.666150] nix-grpc-daemon[770]: ts=2026-09-23T04:44:00Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 98.678119] nix-grpc-daemon[770]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'... worker2 # [ 98.730103] nix-grpc-daemon[773]: ts=2026-09-23T04:44:00Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 98.737582] nix-grpc-daemon[773]: ts=2026-09-23T04:44:00Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_out=232 client # [ 98.804775] nix-daemon[1167]: reaped child process 1169, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.43 seconds) client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.04 seconds) client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds) client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds) (finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.54 seconds) subtest: an input only one worker has still reaches the builder client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only client # this derivation will be built: client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.30 seconds) client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.12 seconds) worker1: must succeed: nix-store --import < /tmp/shared/inp.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.20 seconds) worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.03 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2 client # [ 99.786664] nix-daemon[1167]: accepted connection from pid 1250, user root (trusted) client # this derivation will be built: client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 99.956348] nix-grpc-daemon[770]: ts=2026-09-23T04:44:01Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:59106 worker2 # [ 99.973419] nix-grpc-daemon[773]: ts=2026-09-23T04:44:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 client # copying 1 paths... client # copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' to 'grpc://lb:50051'... worker2 # [ 99.990692] nix-grpc-daemon[773]: ts=2026-09-23T04:44:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 100.006272] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 100.175944] nix-daemon[843]: accepted connection from pid 1198, user nix-grpc-daemon (trusted) worker2 # [ 100.199782] nix-daemon[843]: reaped child process 1201, status = succeeded lb # [ 100.159750] niks3-server[1001]: 2026/09/23 04:44:01 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 100.217189] nix-grpc-daemon[856]: time=2026-09-23T04:44:01.735Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 100.219824] nix-grpc-daemon[856]: time=2026-09-23T04:44:01.737Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)" lb # [ 100.191702] envoy[891]: conn peer=[2001:db8:1::1]:38420 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1443 tx=1197 ms=7034 lb # [ 100.206595] envoy[891]: conn peer=[2001:db8:1::1]:57584 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3414 tx=1943 ms=2141 worker2 # [ 100.265443] nix-grpc-daemon[856]: time=2026-09-23T04:44:01.783Z level=INFO msg="Uploading 1 narinfos" lb # [ 100.223322] envoy[891]: conn peer=[2001:db8:1::1]:57596 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=333 ms=67 lb # [ 100.236773] envoy[891]: conn peer=[2001:db8:1::1]:57598 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3871 tx=2141 ms=1536 lb # [ 100.249644] envoy[891]: conn peer=[2001:db8:1::1]:57602 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=375 ms=31 lb # [ 100.259335] envoy[891]: conn peer=[2001:db8:1::1]:57608 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4339 tx=2392 ms=2642 worker2 # [ 100.316508] nix-grpc-daemon[856]: time=2026-09-23T04:44:01.834Z level=INFO msg="Upload complete. (292ms)" lb # [ 100.273297] envoy[891]: conn peer=[2001:db8:1::1]:57612 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=412 tx=406 ms=29 lb # [ 100.278908] niks3-server[1001]: 2026/09/23 04:44:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign lb # [ 100.285098] niks3-server[1001]: 2026/09/23 04:44:01 INFO Signed narinfos id=27 count=1 lb # [ 100.287423] niks3-server[1001]: 2026/09/23 04:44:01 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls worker2 # [ 100.328468] nix-grpc-daemon[773]: ts=2026-09-23T04:44:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_in=120 lb # [ 100.291641] niks3-server[1001]: 2026/09/23 04:44:01 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst lb # [ 100.297657] niks3-server[1001]: 2026/09/23 04:44:01 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete lb # [ 100.302701] niks3-server[1001]: 2026/09/23 04:44:01 INFO Completed upload id=27 worker2 # [ 100.350503] nix-daemon[843]: reaped child process 1196, status = succeeded lb # [ 100.305333] niks3-server[1001]: 2026/09/23 04:44:01 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo worker1 # [ 100.341623] nix-grpc-daemon[770]: ts=2026-09-23T04:44:01Z level=debug event=want client=14 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=0 worker1 # [ 100.350785] nix-grpc-daemon[770]: ts=2026-09-23T04:44:01Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 100.390660] nix-grpc-daemon[770]: ts=2026-09-23T04:44:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 100.412823] nix-grpc-daemon[770]: ts=2026-09-23T04:44:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59106 client # copying 2 paths to 'lb:50051'... client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'... worker1 # [ 100.439190] nix-grpc-daemon[770]: ts=2026-09-23T04:44:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'... worker1 # [ 100.460772] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 100.627002] nix-daemon[855]: accepted connection from pid 1140, user nix-grpc-daemon (trusted) worker1 # [ 100.648244] nix-daemon[855]: reaped child process 1143, status = succeeded lb # [ 100.619692] niks3-server[1001]: 2026/09/23 04:44:02 INFO Received uploads request method=POST path=/api/pending_closures lb # [ 100.630051] niks3-server[1001]: 2026/09/23 04:44:02 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 100.672535] nix-grpc-daemon[906]: time=2026-09-23T04:44:02.191Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker1 # [ 100.674495] nix-grpc-daemon[906]: time=2026-09-23T04:44:02.191Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)" worker1 # [ 100.680509] nix-grpc-daemon[906]: time=2026-09-23T04:44:02.191Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)" lb # [ 100.681045] niks3-server[1001]: 2026/09/23 04:44:02 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst lb # [ 100.692368] niks3-server[1001]: 2026/09/23 04:44:02 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst lb # [ 100.706758] niks3-server[1001]: 2026/09/23 04:44:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign worker1 # [ 100.740148] nix-grpc-daemon[906]: time=2026-09-23T04:44:02.258Z level=INFO msg="Uploading 3 narinfos" lb # [ 100.713637] niks3-server[1001]: 2026/09/23 04:44:02 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls lb # [ 100.719200] niks3-server[1001]: 2026/09/23 04:44:02 INFO Signed narinfos id=29 count=2 lb # [ 100.723266] niks3-server[1001]: 2026/09/23 04:44:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign lb # [ 100.731325] niks3-server[1001]: 2026/09/23 04:44:02 INFO Signed narinfos id=28 count=1 lb # [ 100.735401] niks3-server[1001]: 2026/09/23 04:44:02 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls lb # [ 100.745544] niks3-server[1001]: 2026/09/23 04:44:02 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo lb # [ 100.755164] niks3-server[1001]: 2026/09/23 04:44:02 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete lb # [ 100.763143] niks3-server[1001]: 2026/09/23 04:44:02 INFO Completed upload id=28 worker1 # [ 100.795792] nix-grpc-daemon[906]: time=2026-09-23T04:44:02.315Z level=INFO msg="Upload complete. (313ms)" lb # [ 100.765065] niks3-server[1001]: 2026/09/23 04:44:02 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete lb # [ 100.768270] niks3-server[1001]: 2026/09/23 04:44:02 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo lb # [ 100.771916] niks3-server[1001]: 2026/09/23 04:44:02 INFO Completed upload id=29 lb # [ 100.774795] niks3-server[1001]: 2026/09/23 04:44:02 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo worker1 # [ 100.807899] nix-grpc-daemon[770]: ts=2026-09-23T04:44:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=0 paths=2 nar_bytes_in=984 worker1 # [ 100.824673] nix-grpc-daemon[770]: ts=2026-09-23T04:44:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 100.835165] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 100.843527] nix-daemon[855]: reaped child process 1138, status = succeeded client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051... worker1 # [ 100.857049] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) client # uses-input-w1onlya> node-a: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 101.936288] nix-daemon[855]: reaped child process 1147, status = succeeded worker1 # [ 102.027158] nix-daemon[855]: accepted connection from pid 1163, user nix-grpc-daemon (trusted) worker1 # [ 102.047649] nix-daemon[855]: reaped child process 1166, status = succeeded lb # [ 102.014836] niks3-server[1001]: 2026/09/23 04:44:03 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 102.060129] nix-grpc-daemon[906]: time=2026-09-23T04:44:03.576Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 102.043480] niks3-server[1001]: 2026/09/23 04:44:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign worker1 # [ 102.084689] nix-grpc-daemon[906]: time=2026-09-23T04:44:03.604Z level=INFO msg="Uploading 1 narinfos" lb # [ 102.054817] niks3-server[1001]: 2026/09/23 04:44:03 INFO Signed narinfos id=30 count=1 lb # [ 102.066084] niks3-server[1001]: 2026/09/23 04:44:03 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv lb # [ 102.080623] niks3-server[1001]: 2026/09/23 04:44:03 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls worker1 # [ 102.119693] nix-grpc-daemon[906]: time=2026-09-23T04:44:03.639Z level=INFO msg="Upload complete. (247ms)" lb # [ 102.088041] niks3-server[1001]: 2026/09/23 04:44:03 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete lb # [ 102.093603] niks3-server[1001]: 2026/09/23 04:44:03 INFO Completed upload id=30 lb # [ 102.097428] niks3-server[1001]: 2026/09/23 04:44:03 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo worker1 # [ 102.132923] nix-grpc-daemon[770]: ts=2026-09-23T04:44:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=15 outputs=1 worker1 # [ 102.146084] nix-daemon[855]: reaped child process 1145, status = succeeded client # copying 1 paths... worker1 # [ 102.178732] nix-grpc-daemon[770]: ts=2026-09-23T04:44:03Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59106 client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'... worker2 # [ 102.227841] nix-grpc-daemon[773]: ts=2026-09-23T04:44:03Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 102.241549] nix-grpc-daemon[773]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'... worker2 # [ 102.258121] nix-grpc-daemon[773]: ts=2026-09-23T04:44:03Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_out=120 client # [ 102.374977] nix-daemon[1167]: reaped child process 1253, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.79 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2 client # [ 102.541187] nix-daemon[1167]: accepted connection from pid 1272, user root (trusted) client # this derivation will be built: client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker2 # [ 102.716040] nix-grpc-daemon[773]: ts=2026-09-23T04:44:04Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:32768 client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 102.708801] nix-grpc-daemon[770]: ts=2026-09-23T04:44:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 102.719329] nix-grpc-daemon[906]: time=2026-09-23T04:44:04.238Z level=INFO msg="All 1 paths already cached" worker1 # [ 102.737536] nix-grpc-daemon[770]: ts=2026-09-23T04:44:04Z level=debug event=want client=15 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=0 worker1 # [ 102.744405] nix-grpc-daemon[770]: ts=2026-09-23T04:44:04Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 102.759881] nix-grpc-daemon[773]: ts=2026-09-23T04:44:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 102.776562] nix-grpc-daemon[773]: ts=2026-09-23T04:44:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'... worker2 # [ 102.799853] nix-grpc-daemon[773]: ts=2026-09-23T04:44:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 102.815023] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 102.837746] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 68 ms (attempt 1/5) worker2 # [ 102.915133] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 1 ms (attempt 2/5) worker2 # [ 102.924643] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 154 ms (attempt 3/5) worker2 # [ 103.080416] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 429 ms (attempt 4/5) worker2 # [ 103.513303] nix-grpc-daemon[773]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 103.521363] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 103.528872] nix-grpc-daemon[773]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'... worker2 # [ 103.693936] nix-daemon[843]: accepted connection from pid 1211, user nix-grpc-daemon (trusted) worker2 # [ 103.720376] nix-daemon[843]: reaped child process 1214, status = succeeded lb # [ 103.674736] niks3-server[1001]: 2026/09/23 04:44:05 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 103.729430] nix-grpc-daemon[856]: time=2026-09-23T04:44:05.247Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 103.731019] nix-grpc-daemon[856]: time=2026-09-23T04:44:05.247Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)" lb # [ 103.702804] niks3-server[1001]: 2026/09/23 04:44:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign worker2 # [ 103.756704] nix-grpc-daemon[856]: time=2026-09-23T04:44:05.275Z level=INFO msg="Uploading 1 narinfos" lb # [ 103.716947] niks3-server[1001]: 2026/09/23 04:44:05 INFO Signed narinfos id=31 count=1 lb # [ 103.724509] niks3-server[1001]: 2026/09/23 04:44:05 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst lb # [ 103.738411] niks3-server[1001]: 2026/09/23 04:44:05 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete worker2 # [ 103.792964] nix-grpc-daemon[856]: time=2026-09-23T04:44:05.311Z level=INFO msg="Upload complete. (251ms)" lb # [ 103.749219] niks3-server[1001]: 2026/09/23 04:44:05 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls lb # [ 103.754978] niks3-server[1001]: 2026/09/23 04:44:05 INFO Completed upload id=31 lb # [ 103.758926] niks3-server[1001]: 2026/09/23 04:44:05 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo worker2 # [ 103.803744] nix-grpc-daemon[773]: ts=2026-09-23T04:44:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_in=568 worker2 # [ 103.812320] nix-grpc-daemon[773]: ts=2026-09-23T04:44:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 103.824806] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 103.830127] nix-daemon[843]: reaped child process 1204, status = succeeded client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051... worker2 # [ 103.841573] nix-daemon[843]: accepted connection from pid 773, user nix-grpc-daemon (trusted) client # uses-input-w1onlyb> worker2: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 105.302961] nix-daemon[843]: reaped child process 1218, status = succeeded worker2 # [ 105.431512] nix-daemon[843]: accepted connection from pid 1234, user nix-grpc-daemon (trusted) worker2 # [ 105.454605] nix-daemon[843]: reaped child process 1237, status = succeeded lb # [ 105.409364] niks3-server[1001]: 2026/09/23 04:44:06 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 105.464520] nix-grpc-daemon[856]: time=2026-09-23T04:44:06.981Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 105.439230] niks3-server[1001]: 2026/09/23 04:44:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign worker2 # [ 105.493881] nix-grpc-daemon[856]: time=2026-09-23T04:44:07.012Z level=INFO msg="Uploading 1 narinfos" lb # [ 105.455039] niks3-server[1001]: 2026/09/23 04:44:06 INFO Signed narinfos id=32 count=1 lb # [ 105.462378] niks3-server[1001]: 2026/09/23 04:44:06 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls worker2 # [ 105.517868] nix-grpc-daemon[856]: time=2026-09-23T04:44:07.036Z level=INFO msg="Upload complete. (266ms)" lb # [ 105.472657] niks3-server[1001]: 2026/09/23 04:44:06 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv lb # [ 105.479370] niks3-server[1001]: 2026/09/23 04:44:06 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete lb # [ 105.486443] niks3-server[1001]: 2026/09/23 04:44:07 INFO Completed upload id=32 worker2 # [ 105.527444] nix-grpc-daemon[773]: ts=2026-09-23T04:44:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=16 outputs=1 lb # [ 105.488679] niks3-server[1001]: 2026/09/23 04:44:07 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo worker2 # [ 105.545567] nix-daemon[843]: reaped child process 1216, status = succeeded client # copying 1 paths... worker1 # [ 105.563192] nix-grpc-daemon[770]: ts=2026-09-23T04:44:07Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 105.577038] nix-grpc-daemon[770]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'... client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'... worker1 # [ 105.621866] nix-grpc-daemon[770]: ts=2026-09-23T04:44:07Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 105.630799] nix-grpc-daemon[770]: ts=2026-09-23T04:44:07Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=0 paths=1 nar_bytes_out=120 client # [ 105.750427] nix-daemon[1167]: reaped child process 1275, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.37 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2 client # [ 105.923474] nix-daemon[1167]: accepted connection from pid 1294, user root (trusted) client # this derivation will be built: client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker2 # [ 106.094399] nix-grpc-daemon[773]: ts=2026-09-23T04:44:07Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:32768 client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker2 # [ 106.102258] nix-grpc-daemon[773]: ts=2026-09-23T04:44:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 106.109489] nix-grpc-daemon[856]: time=2026-09-23T04:44:07.626Z level=INFO msg="All 1 paths already cached" worker1 # [ 106.109693] nix-grpc-daemon[770]: ts=2026-09-23T04:44:07Z level=debug event=want client=16 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=0 worker1 # [ 106.118038] nix-grpc-daemon[770]: ts=2026-09-23T04:44:07Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 106.127582] nix-grpc-daemon[770]: ts=2026-09-23T04:44:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 106.139016] nix-grpc-daemon[770]: ts=2026-09-23T04:44:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59106 client # copying 1 paths to 'lb:50051'... worker1 # [ 106.155657] nix-grpc-daemon[906]: time=2026-09-23T04:44:07.672Z level=INFO msg="All 1 paths already cached" client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'... worker1 # [ 106.161798] nix-grpc-daemon[770]: ts=2026-09-23T04:44:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 106.177096] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 106.326839] nix-daemon[855]: accepted connection from pid 1175, user nix-grpc-daemon (trusted) worker1 # [ 106.348466] nix-daemon[855]: reaped child process 1178, status = succeeded lb # [ 106.314088] niks3-server[1001]: 2026/09/23 04:44:07 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 106.356183] nix-grpc-daemon[906]: time=2026-09-23T04:44:07.875Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 106.357680] nix-grpc-daemon[906]: time=2026-09-23T04:44:07.875Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)" lb # [ 106.349604] niks3-server[1001]: 2026/09/23 04:44:07 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst worker1 # [ 106.394268] nix-grpc-daemon[906]: time=2026-09-23T04:44:07.913Z level=INFO msg="Uploading 1 narinfos" lb # [ 106.364902] niks3-server[1001]: 2026/09/23 04:44:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign lb # [ 106.374240] niks3-server[1001]: 2026/09/23 04:44:07 INFO Signed narinfos id=33 count=1 lb # [ 106.382094] niks3-server[1001]: 2026/09/23 04:44:07 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls worker1 # [ 106.419627] nix-grpc-daemon[906]: time=2026-09-23T04:44:07.939Z level=INFO msg="Upload complete. (227ms)" lb # [ 106.391011] niks3-server[1001]: 2026/09/23 04:44:07 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete lb # [ 106.399035] niks3-server[1001]: 2026/09/23 04:44:07 INFO Completed upload id=33 worker1 # [ 106.428321] nix-grpc-daemon[770]: ts=2026-09-23T04:44:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=0 paths=1 nar_bytes_in=568 lb # [ 106.402895] niks3-server[1001]: 2026/09/23 04:44:07 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo worker1 # [ 106.444597] nix-grpc-daemon[770]: ts=2026-09-23T04:44:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 106.450702] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) worker1 # [ 106.455584] nix-daemon[855]: reaped child process 1173, status = succeeded client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051... worker1 # [ 106.467781] nix-daemon[855]: accepted connection from pid 770, user nix-grpc-daemon (trusted) client # uses-input-w1onlyc> node-a: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 107.728235] nix-daemon[855]: reaped child process 1182, status = succeeded worker1 # [ 107.822769] nix-daemon[855]: accepted connection from pid 1198, user nix-grpc-daemon (trusted) worker1 # [ 107.841817] nix-daemon[855]: reaped child process 1201, status = succeeded lb # [ 107.815361] niks3-server[1001]: 2026/09/23 04:44:09 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 107.858734] nix-grpc-daemon[906]: time=2026-09-23T04:44:09.376Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 107.847145] niks3-server[1001]: 2026/09/23 04:44:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign worker1 # [ 107.890607] nix-grpc-daemon[906]: time=2026-09-23T04:44:09.409Z level=INFO msg="Uploading 1 narinfos" lb # [ 107.864304] niks3-server[1001]: 2026/09/23 04:44:09 INFO Signed narinfos id=34 count=1 lb # [ 107.874398] niks3-server[1001]: 2026/09/23 04:44:09 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv lb # [ 107.883368] niks3-server[1001]: 2026/09/23 04:44:09 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls lb # [ 107.891595] niks3-server[1001]: 2026/09/23 04:44:09 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete worker1 # [ 107.931250] nix-grpc-daemon[906]: time=2026-09-23T04:44:09.450Z level=INFO msg="Upload complete. (249ms)" lb # [ 107.898848] niks3-server[1001]: 2026/09/23 04:44:09 INFO Completed upload id=34 lb # [ 107.903637] niks3-server[1001]: 2026/09/23 04:44:09 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo worker1 # [ 107.945127] nix-grpc-daemon[770]: ts=2026-09-23T04:44:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59106 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=17 outputs=1 worker1 # [ 107.960959] nix-daemon[855]: reaped child process 1180, status = succeeded client # copying 1 paths... worker2 # [ 108.003973] nix-grpc-daemon[773]: ts=2026-09-23T04:44:09Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 108.013907] nix-grpc-daemon[773]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'... client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'... worker2 # [ 108.060176] nix-grpc-daemon[773]: ts=2026-09-23T04:44:09Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 worker2 # [ 108.067102] nix-grpc-daemon[773]: ts=2026-09-23T04:44:09Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:32768 duration_s=0 paths=1 nar_bytes_out=120 client # [ 108.174122] nix-daemon[1167]: reaped child process 1297, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.42 seconds) (finished: subtest: an input only one worker has still reaches the builder, in 9.24 seconds) subtest: upload whose reference the worker lacks is completed from the cache client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache client # this derivation will be built: client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.34 seconds) client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache client # this derivation will be built: client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.30 seconds) client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.13 seconds) worker1: must succeed: nix-store --import < /tmp/shared/ref.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.20 seconds) worker2 # [ 109.239028] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 109.609336] nix-grpc-daemon[773]: ts=2026-09-23T04:44:11Z level=info event=signal_exit lb # [ 110.188705] envoy[891]: conn peer=[2001:db8:1::1]:42618 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=329 ms=97 lb # [ 110.193704] envoy[891]: conn peer=[2001:db8:1::1]:57622 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4271 tx=2050 ms=2352 lb # [ 110.198554] envoy[891]: conn peer=[2001:db8:1::1]:42622 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3766 tx=2127 ms=2986 lb # [ 110.204665] envoy[891]: conn peer=[2001:db8:1::1]:42626 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=72 lb # [ 110.209443] envoy[891]: conn peer=[2001:db8:1::1]:42628 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3737 tx=2083 ms=2031 lb # [ 110.214282] envoy[891]: conn peer=[2001:db8:1::1]:42644 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=71 worker1 # [ 110.675801] nix-grpc-daemon[770]: ts=2026-09-23T04:44:12Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:45280 worker2 # [ 110.693032] nix-grpc-daemon[773]: ts=2026-09-23T04:44:12Z level=info event=scheduler_restarting worker1 # [ 110.680777] nix-grpc-daemon[770]: ts=2026-09-23T04:44:12Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 110.720529] nix-grpc-daemon[773]: ts=2026-09-23T04:44:12Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 110.740256] nix-daemon[843]: reaped child process 845, status = succeeded worker2 # [ 110.802087] nix-grpc-daemon[773]: E0000 00:00:1790138652.320292 819 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true worker2 # [ 110.808786] nix-grpc-daemon[773]: E0000 00:00:1790138652.320640 819 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker2 # [ 110.929149] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 110.939721] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 110.944980] systemd[1]: nix-grpc-daemon.service: Consumed 3.904s CPU time over 1min 30.571s wall clock time, 65.6M memory peak, 111.8K incoming IP traffic, 262.2K outgoing IP traffic. worker2 # [ 110.958362] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker2 # [ 110.961738] systemd[1]: Closed Nix worker-protocol over gRPC. worker2 # [ 110.963528] systemd[1]: nix-grpc-daemon.socket: Consumed 682us CPU time over 1min 40.436s wall clock time, 256K memory peak, 106.1K incoming IP traffic, 90.8K outgoing IP traffic. lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds) client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2 worker1 # [ 111.255620] nix-grpc-daemon[770]: ts=2026-09-23T04:44:12Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59106 worker1 # [ 111.262399] nix-grpc-daemon[770]: ts=2026-09-23T04:44:12Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59106 client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.25 seconds) worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds) worker2 # [ 111.398162] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 111.457497] systemd[1]: Stopping Nix worker-protocol over gRPC... worker1 # [ 111.474256] nix-grpc-daemon[770]: ts=2026-09-23T04:44:12Z level=info event=signal_exit lb # [ 111.638044] niks3-server[1001]: 2026/09/23 04:44:13 INFO lead: released remote=192.168.1.3:38116 worker2 # [ 111.931001] systemd[1]: Starting scheduler-token.service... worker2 # [ 112.018780] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 112.028717] systemd[1]: Finished scheduler-token.service. worker2 # [ 112.030821] systemd[1]: scheduler-token.service: Consumed 58ms CPU time over 92ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 112.039275] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 112.173601] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:13Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 112.175893] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:13Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 112.179957] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:13Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 112.186990] nix-grpc-daemon[1262]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 112.203907] nix-grpc-daemon[1262]: E0000 00:00:1790138653.703122 1262 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 112.160179] niks3-server[1001]: 2026/09/23 04:44:13 INFO lead: acquired remote=192.168.1.4:59238 worker2 # [ 112.235691] nix-grpc-daemon[1262]: E0000 00:00:1790138653.705302 1262 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 112.239114] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:13Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 112.241478] nix-grpc-daemon[1262]: E0000 00:00:1790138653.717887 1302 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 112.243994] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:13Z level=info event=scheduler_take_over worker2 # [ 112.245140] nix-grpc-daemon[1262]: E0000 00:00:1790138653.739189 1302 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 112.250283] nix-grpc-daemon[1262]: E0000 00:00:1790138653.739372 1302 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 112.255183] nix-grpc-daemon[1262]: E0000 00:00:1790138653.754076 1302 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 112.257787] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 112.258621] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:13Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 112.259866] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 112.456738] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:13Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 112.461762] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 112.662912] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:14Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 112.668279] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 112.671779] nix-grpc-daemon[770]: ts=2026-09-23T04:44:14Z level=info event=scheduler_restarting worker1 # [ 112.682171] nix-grpc-daemon[770]: ts=2026-09-23T04:44:14Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 112.686826] nix-grpc-daemon[770]: ts=2026-09-23T04:44:14Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:45280 worker1 # [ 112.691876] nix-grpc-daemon[770]: ts=2026-09-23T04:44:14Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 112.719825] nix-daemon[855]: reaped child process 857, status = succeeded worker2 # [ 112.869283] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:14Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 112.874410] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 112.897607] nix-grpc-daemon[770]: ts=2026-09-23T04:44:14Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 113.075538] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:14Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 113.080911] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 113.283267] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:14Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 113.300173] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:14Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 113.304544] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:14Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:35044 worker2 # [ 113.309822] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:14Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 113.642001] nix-grpc-daemon[770]: E0000 00:00:1790138655.160986 816 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true worker1 # [ 113.642448] nix-grpc-daemon[770]: E0000 00:00:1790138655.161394 816 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker1 # [ 114.872250] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 114.876667] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 114.883905] systemd[1]: nix-grpc-daemon.service: Consumed 2.893s CPU time over 1min 34.659s wall clock time, 68.4M memory peak, 97.7K incoming IP traffic, 193.1K outgoing IP traffic. worker1 # [ 114.900634] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker1 # [ 114.902943] systemd[1]: Closed Nix worker-protocol over gRPC. worker1 # [ 114.909420] systemd[1]: nix-grpc-daemon.socket: Consumed 581us CPU time over 1min 44.314s wall clock time, 256K memory peak, 118.9K incoming IP traffic, 101.1K outgoing IP traffic. lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds) client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2 worker2 # [ 115.229242] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:35050 worker2 # [ 115.244794] nix-daemon[843]: accepted connection from pid 1262, user nix-grpc-daemon (trusted) worker2 # [ 115.274937] nix-grpc-daemon[1262]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 17 ms (attempt 1/5) worker2 # [ 115.299973] nix-grpc-daemon[1262]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 186 ms (attempt 2/5) worker2 # [ 115.487998] nix-grpc-daemon[1262]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 3 ms (attempt 3/5) worker2 # [ 115.497298] nix-grpc-daemon[1262]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 665 ms (attempt 4/5) worker2 # [ 116.164453] nix-grpc-daemon[1262]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 116.168729] nix-grpc-daemon[1262]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org client # copying 2 paths... client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'... worker2 # [ 116.196136] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35050 client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'... worker2 # [ 116.213994] nix-daemon[843]: accepted connection from pid 1262, user nix-grpc-daemon (trusted) worker2 # [ 116.245810] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:17Z level=info event=niks3_push_started pid=1323 worker2 # [ 116.400585] nix-grpc-daemon[1323]: time=2026-09-23T04:44:17.918Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 116.511010] nix-daemon[843]: accepted connection from pid 1336, user nix-grpc-daemon (trusted) worker2 # [ 116.530679] nix-daemon[843]: reaped child process 1339, status = succeeded lb # [ 116.491128] niks3-server[1001]: 2026/09/23 04:44:18 INFO Received uploads request method=POST path=/api/pending_closures lb # [ 116.682426] niks3-server[1001]: 2026/09/23 04:44:18 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 116.740155] nix-grpc-daemon[1323]: time=2026-09-23T04:44:18.257Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker2 # [ 116.754250] nix-grpc-daemon[1323]: time=2026-09-23T04:44:18.257Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)" worker2 # [ 116.760183] nix-grpc-daemon[1323]: time=2026-09-23T04:44:18.257Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)" lb # [ 116.766728] niks3-server[1001]: 2026/09/23 04:44:18 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls lb # [ 116.778970] niks3-server[1001]: 2026/09/23 04:44:18 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst worker2 # [ 116.826628] nix-grpc-daemon[1323]: time=2026-09-23T04:44:18.342Z level=INFO msg="Uploading 3 narinfos" lb # [ 116.790354] niks3-server[1001]: 2026/09/23 04:44:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign lb # [ 116.797100] niks3-server[1001]: 2026/09/23 04:44:18 INFO Signed narinfos id=36 count=2 lb # [ 116.799301] niks3-server[1001]: 2026/09/23 04:44:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign lb # [ 116.808239] niks3-server[1001]: 2026/09/23 04:44:18 INFO Signed narinfos id=35 count=1 lb # [ 116.810412] niks3-server[1001]: 2026/09/23 04:44:18 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst lb # [ 116.823481] niks3-server[1001]: 2026/09/23 04:44:18 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls lb # [ 116.829801] niks3-server[1001]: 2026/09/23 04:44:18 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo lb # [ 116.837443] niks3-server[1001]: 2026/09/23 04:44:18 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete lb # [ 116.843136] niks3-server[1001]: 2026/09/23 04:44:18 INFO Completed upload id=35 lb # [ 116.848260] niks3-server[1001]: 2026/09/23 04:44:18 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete lb # [ 116.856786] niks3-server[1001]: 2026/09/23 04:44:18 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo worker2 # [ 116.899314] nix-grpc-daemon[1323]: time=2026-09-23T04:44:18.417Z level=INFO msg="Upload complete. (494ms)" lb # [ 116.859004] niks3-server[1001]: 2026/09/23 04:44:18 INFO Completed upload id=36 worker2 # [ 116.912326] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35050 duration_s=0 paths=2 nar_bytes_in=312 lb # [ 116.873954] niks3-server[1001]: 2026/09/23 04:44:18 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo worker2 # [ 116.929501] nix-daemon[843]: reaped child process 1321, status = succeeded client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.94 seconds) worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds) worker1 # [ 117.041081] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 117.048427] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 117.189219] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:18Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 117.191543] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:18Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 117.195475] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:18Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 117.202680] nix-grpc-daemon[1224]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 117.210894] nix-grpc-daemon[1224]: E0000 00:00:1790138658.719900 1224 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 117.215459] nix-grpc-daemon[1224]: E0000 00:00:1790138658.722015 1224 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 117.248189] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:18Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 117.249995] nix-grpc-daemon[1224]: E0000 00:00:1790138658.734112 1264 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 117.255502] nix-grpc-daemon[1224]: E0000 00:00:1790138658.753142 1264 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 117.258482] nix-grpc-daemon[1224]: E0000 00:00:1790138658.753308 1264 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 117.261103] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 117.261969] nix-grpc-daemon[1224]: E0000 00:00:1790138658.773954 1264 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 117.275143] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 117.292504] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:18Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:35044 worker2 # [ 117.297399] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:18Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.14 seconds) (finished: subtest: upload whose reference the worker lacks is completed from the cache, in 10.19 seconds) subtest: a niks3 restart does not cost the leader its role lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb # [ 118.475502] niks3-server[1001]: 2026/09/23 04:44:20 INFO Shutdown signal received, draining in-flight requests timeout=10s worker1 # [ 118.508922] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:20Z level=info event=scheduler_lock_lost status=200 worker2 # [ 118.521256] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:20Z level=info event=scheduler_lock_lost status=200 lb # [ 118.481973] niks3-server[1001]: 2026/09/23 04:44:20 INFO lead: released remote=192.168.1.4:59238 lb # [ 118.495194] systemd[1]: Stopping niks3 server... lb # [ 118.504536] systemd[1]: niks3.service: Deactivated successfully. lb # [ 118.507075] systemd[1]: Stopped niks3 server. lb # [ 118.507801] systemd[1]: niks3.service: Consumed 678ms CPU time over 1min 29.173s wall clock time, 27.7M memory peak, 95.8K incoming IP traffic, 139.5K outgoing IP traffic. lb # [ 118.517628] systemd[1]: Starting niks3 server... lb # [ 118.575119] niks3-server[1141]: 2026/09/23 04:44:20 goose: no migrations to run. current version: 20260920000000 lb # [ 118.579994] niks3-server[1141]: 2026/09/23 04:44:20 OK 1_commit_pending_closure.sql (6.54ms) lb # [ 118.583612] niks3-server[1141]: 2026/09/23 04:44:20 OK 2_object_stats_trigger.sql (3.26ms) lb # [ 118.585526] niks3-server[1141]: 2026/09/23 04:44:20 goose: up to current file version: 2 lb # [ 118.587699] niks3-server[1141]: 2026/09/23 04:44:20 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key lb # [ 118.597913] niks3-server[1141]: 2026/09/23 04:44:20 INFO Read proxy enabled — serving cache objects from S3 lb # [ 118.599243] niks3-server[1141]: 2026/09/23 04:44:20 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 118.604715] niks3-server[1141]: 2026/09/23 04:44:20 INFO systemd watchdog enabled interval=15s lb # [ 118.606517] niks3-server[1141]: 2026/09/23 04:44:20 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 118.608293] systemd[1]: Started niks3 server. lb: waiting for unit niks3.service lb: (finished: waiting for unit niks3.service, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.07 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2 worker1 # [ 119.124136] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:20Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 119.140188] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted) client # don't know how to build these paths: client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker2 # [ 119.176621] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:35050 worker2 # [ 119.209962] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:20Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=0 worker2 # [ 119.217528] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:20Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 119.256707] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35050 worker2 # [ 119.272737] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:35050 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'... worker2 # [ 119.288721] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35050 worker2 # [ 119.303413] nix-daemon[843]: accepted connection from pid 1262, user nix-grpc-daemon (trusted) worker2 # [ 119.457045] nix-daemon[843]: accepted connection from pid 1356, user nix-grpc-daemon (trusted) worker2 # [ 119.476771] nix-daemon[843]: reaped child process 1359, status = succeeded lb # [ 119.436926] niks3-server[1141]: 2026/09/23 04:44:20 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 119.495572] nix-grpc-daemon[1323]: time=2026-09-23T04:44:21.013Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 119.497381] nix-grpc-daemon[1323]: time=2026-09-23T04:44:21.013Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)" lb # [ 119.485676] niks3-server[1141]: 2026/09/23 04:44:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign lb # [ 119.509429] niks3-server[1141]: 2026/09/23 04:44:21 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls lb # [ 119.539473] niks3-server[1141]: 2026/09/23 04:44:21 INFO Signed narinfos id=37 count=1 worker2 # [ 119.587891] nix-grpc-daemon[1323]: time=2026-09-23T04:44:21.105Z level=INFO msg="Uploading 1 narinfos" lb # [ 119.545388] niks3-server[1141]: 2026/09/23 04:44:21 INFO lead: acquired remote=192.168.1.4:59336 lb # [ 119.556883] niks3-server[1141]: 2026/09/23 04:44:21 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst lb # [ 119.568388] niks3-server[1141]: 2026/09/23 04:44:21 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete worker2 # [ 119.620717] nix-grpc-daemon[1323]: time=2026-09-23T04:44:21.139Z level=INFO msg="Upload complete. (300ms)" lb # [ 119.581070] niks3-server[1141]: 2026/09/23 04:44:21 INFO Completed upload id=37 lb # [ 119.584328] niks3-server[1141]: 2026/09/23 04:44:21 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo worker2 # [ 119.634320] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35050 duration_s=0 paths=1 nar_bytes_in=544 worker2 # [ 119.648808] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35050 worker2 # [ 119.660064] nix-daemon[843]: accepted connection from pid 1262, user nix-grpc-daemon (trusted) worker2 # [ 119.666942] nix-daemon[843]: reaped child process 1354, status = succeeded client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051... worker2 # [ 119.678062] nix-daemon[843]: accepted connection from pid 1262, user nix-grpc-daemon (trusted) client # farm-a> worker2: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv lb # [ 120.191961] envoy[891]: conn peer=[2001:db8:1::4]:40330 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=76951 tx=4359 ms=90071 lb # [ 120.197077] envoy[891]: conn peer=[2001:db8:1::1]:46352 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=523 ms=28 lb # [ 120.201697] envoy[891]: conn peer=[2001:db8:1::3]:32914 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=15224 tx=4184 ms=92379 lb # [ 120.206613] envoy[891]: conn peer=[2001:db8:1::1]:46362 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1006 tx=304 ms=1700 client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-after-niks3-restart worker2 # [ 121.352997] nix-daemon[843]: reaped child process 1365, status = succeeded worker2 # [ 121.478159] nix-daemon[843]: accepted connection from pid 1381, user nix-grpc-daemon (trusted) worker2 # [ 121.498389] nix-daemon[843]: reaped child process 1384, status = succeeded lb # [ 121.455963] niks3-server[1141]: 2026/09/23 04:44:22 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 121.515311] nix-grpc-daemon[1323]: time=2026-09-23T04:44:23.031Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 121.517109] nix-grpc-daemon[1323]: time=2026-09-23T04:44:23.031Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)" lb # [ 121.503628] niks3-server[1141]: 2026/09/23 04:44:23 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls worker2 # [ 121.562085] nix-grpc-daemon[1323]: time=2026-09-23T04:44:23.080Z level=INFO msg="Uploading 1 narinfos" lb # [ 121.518416] niks3-server[1141]: 2026/09/23 04:44:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign lb # [ 121.529743] niks3-server[1141]: 2026/09/23 04:44:23 INFO Signed narinfos id=38 count=1 lb # [ 121.535576] niks3-server[1141]: 2026/09/23 04:44:23 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv lb # [ 121.546950] niks3-server[1141]: 2026/09/23 04:44:23 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst lb # [ 121.552284] niks3-server[1141]: 2026/09/23 04:44:23 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete worker2 # [ 121.596511] nix-grpc-daemon[1323]: time=2026-09-23T04:44:23.114Z level=INFO msg="Upload complete. (304ms)" lb # [ 121.555340] niks3-server[1141]: 2026/09/23 04:44:23 INFO Completed upload id=38 lb # [ 121.558301] niks3-server[1141]: 2026/09/23 04:44:23 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo worker2 # [ 121.620909] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35050 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=1 worker2 # [ 121.635678] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:23Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=0 worker2 # [ 121.639707] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:23Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 121.643780] nix-daemon[843]: reaped child process 1363, status = succeeded worker2 # [ 121.661107] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35050 worker2 # [ 121.676335] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:35050 worker2 # [ 121.692309] nix-grpc-daemon[1323]: time=2026-09-23T04:44:23.210Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'... worker2 # [ 121.703019] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35050 worker2 # [ 121.716736] nix-daemon[843]: accepted connection from pid 1262, user nix-grpc-daemon (trusted) worker2 # [ 121.870695] nix-daemon[843]: accepted connection from pid 1390, user nix-grpc-daemon (trusted) worker2 # [ 121.889799] nix-daemon[843]: reaped child process 1393, status = succeeded lb # [ 121.849046] niks3-server[1141]: 2026/09/23 04:44:23 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 121.903044] nix-grpc-daemon[1323]: time=2026-09-23T04:44:23.421Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 121.904623] nix-grpc-daemon[1323]: time=2026-09-23T04:44:23.421Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)" lb # [ 121.880317] niks3-server[1141]: 2026/09/23 04:44:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign worker2 # [ 121.935641] nix-grpc-daemon[1323]: time=2026-09-23T04:44:23.453Z level=INFO msg="Uploading 1 narinfos" lb # [ 121.896390] niks3-server[1141]: 2026/09/23 04:44:23 INFO Signed narinfos id=39 count=1 lb # [ 121.899596] niks3-server[1141]: 2026/09/23 04:44:23 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls worker2 # [ 121.959322] nix-grpc-daemon[1323]: time=2026-09-23T04:44:23.477Z level=INFO msg="Upload complete. (227ms)" lb # [ 121.916653] niks3-server[1141]: 2026/09/23 04:44:23 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst lb # [ 121.925860] niks3-server[1141]: 2026/09/23 04:44:23 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete worker2 # [ 121.968206] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35050 duration_s=0 paths=1 nar_bytes_in=656 lb # [ 121.929381] niks3-server[1141]: 2026/09/23 04:44:23 INFO Completed upload id=39 lb # [ 121.931278] niks3-server[1141]: 2026/09/23 04:44:23 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo worker2 # [ 121.980505] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35050 worker2 # [ 121.992177] nix-daemon[843]: accepted connection from pid 1262, user nix-grpc-daemon (trusted) worker2 # [ 121.997589] nix-daemon[843]: reaped child process 1388, status = succeeded client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051... client # farm-b> worker2: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker2 # [ 122.012149] nix-daemon[843]: accepted connection from pid 1262, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-after-niks3-restart worker2 # [ 123.109590] nix-daemon[843]: reaped child process 1397, status = succeeded worker2 # [ 123.174308] nix-daemon[843]: accepted connection from pid 1413, user nix-grpc-daemon (trusted) worker2 # [ 123.195788] nix-daemon[843]: reaped child process 1416, status = succeeded lb # [ 123.155087] niks3-server[1141]: 2026/09/23 04:44:24 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 123.212393] nix-grpc-daemon[1323]: time=2026-09-23T04:44:24.729Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 123.214319] nix-grpc-daemon[1323]: time=2026-09-23T04:44:24.729Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)" lb # [ 123.199531] niks3-server[1141]: 2026/09/23 04:44:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign worker2 # [ 123.252526] nix-grpc-daemon[1323]: time=2026-09-23T04:44:24.770Z level=INFO msg="Uploading 1 narinfos" lb # [ 123.213526] niks3-server[1141]: 2026/09/23 04:44:24 INFO Signed narinfos id=40 count=1 lb # [ 123.223256] niks3-server[1141]: 2026/09/23 04:44:24 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst lb # [ 123.236129] niks3-server[1141]: 2026/09/23 04:44:24 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls lb # [ 123.239382] niks3-server[1141]: 2026/09/23 04:44:24 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv lb # [ 123.252643] niks3-server[1141]: 2026/09/23 04:44:24 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete lb # [ 123.260719] niks3-server[1141]: 2026/09/23 04:44:24 INFO Completed upload id=40 lb # [ 123.263782] niks3-server[1141]: 2026/09/23 04:44:24 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo worker2 # [ 123.296226] nix-grpc-daemon[1323]: time=2026-09-23T04:44:24.814Z level=INFO msg="Upload complete. (273ms)" worker2 # [ 123.313323] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35050 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=1 worker2 # [ 123.332416] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:24Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=0 worker2 # [ 123.339342] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:24Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 123.344260] nix-daemon[843]: reaped child process 1395, status = succeeded worker2 # [ 123.361716] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35050 worker2 # [ 123.376950] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:35050 worker2 # [ 123.391909] nix-grpc-daemon[1323]: time=2026-09-23T04:44:24.910Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'... worker2 # [ 123.403241] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35050 worker2 # [ 123.420183] nix-daemon[843]: accepted connection from pid 1262, user nix-grpc-daemon (trusted) worker2 # [ 123.566154] nix-daemon[843]: accepted connection from pid 1422, user nix-grpc-daemon (trusted) worker2 # [ 123.583776] nix-daemon[843]: reaped child process 1425, status = succeeded lb # [ 123.544580] niks3-server[1141]: 2026/09/23 04:44:25 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 123.599226] nix-grpc-daemon[1323]: time=2026-09-23T04:44:25.117Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 123.600883] nix-grpc-daemon[1323]: time=2026-09-23T04:44:25.117Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)" lb # [ 123.574518] niks3-server[1141]: 2026/09/23 04:44:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign worker2 # [ 123.628716] nix-grpc-daemon[1323]: time=2026-09-23T04:44:25.146Z level=INFO msg="Uploading 1 narinfos" lb # [ 123.590894] niks3-server[1141]: 2026/09/23 04:44:25 INFO Signed narinfos id=41 count=1 lb # [ 123.597177] niks3-server[1141]: 2026/09/23 04:44:25 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst lb # [ 123.608768] niks3-server[1141]: 2026/09/23 04:44:25 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls worker2 # [ 123.658225] nix-grpc-daemon[1323]: time=2026-09-23T04:44:25.176Z level=INFO msg="Upload complete. (226ms)" lb # [ 123.617631] niks3-server[1141]: 2026/09/23 04:44:25 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete lb # [ 123.623212] niks3-server[1141]: 2026/09/23 04:44:25 INFO Completed upload id=41 worker2 # [ 123.668292] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35050 duration_s=0 paths=1 nar_bytes_in=784 lb # [ 123.626914] niks3-server[1141]: 2026/09/23 04:44:25 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo worker2 # [ 123.679961] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35050 worker2 # [ 123.687187] nix-daemon[843]: accepted connection from pid 1262, user nix-grpc-daemon (trusted) worker2 # [ 123.692820] nix-daemon[843]: reaped child process 1420, status = succeeded client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051... worker2 # [ 123.706751] nix-daemon[843]: accepted connection from pid 1262, user nix-grpc-daemon (trusted) client # farm-top> worker2: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-after-niks3-restart worker2 # [ 124.775394] nix-daemon[843]: reaped child process 1429, status = succeeded worker2 # [ 124.905845] nix-daemon[843]: accepted connection from pid 1445, user nix-grpc-daemon (trusted) worker2 # [ 124.926888] nix-daemon[843]: reaped child process 1448, status = succeeded lb # [ 124.885409] niks3-server[1141]: 2026/09/23 04:44:26 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 124.940422] nix-grpc-daemon[1323]: time=2026-09-23T04:44:26.458Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 124.942194] nix-grpc-daemon[1323]: time=2026-09-23T04:44:26.458Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)" lb # [ 124.930272] niks3-server[1141]: 2026/09/23 04:44:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign worker2 # [ 124.984295] nix-grpc-daemon[1323]: time=2026-09-23T04:44:26.502Z level=INFO msg="Uploading 1 narinfos" lb # [ 124.944530] niks3-server[1141]: 2026/09/23 04:44:26 INFO Signed narinfos id=42 count=1 lb # [ 124.954504] niks3-server[1141]: 2026/09/23 04:44:26 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls lb # [ 124.963086] niks3-server[1141]: 2026/09/23 04:44:26 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst lb # [ 124.970113] niks3-server[1141]: 2026/09/23 04:44:26 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv lb # [ 124.979715] niks3-server[1141]: 2026/09/23 04:44:26 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete worker2 # [ 125.023753] nix-grpc-daemon[1323]: time=2026-09-23T04:44:26.542Z level=INFO msg="Upload complete. (303ms)" lb # [ 124.983973] niks3-server[1141]: 2026/09/23 04:44:26 INFO Completed upload id=42 lb # [ 124.987042] niks3-server[1141]: 2026/09/23 04:44:26 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo worker2 # [ 125.037276] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35050 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=1 worker2 # [ 125.058391] nix-daemon[843]: reaped child process 1427, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2, in 6.18 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath, in 0.18 seconds) (finished: subtest: a niks3 restart does not cost the leader its role, in 6.87 seconds) subtest: scheduler down: the other node takes the lock and keeps it lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2 # [ 125.513146] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 126.199191] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:27Z level=info event=signal_exit lb # [ 126.499498] niks3-server[1141]: 2026/09/23 04:44:28 INFO lead: released remote=192.168.1.4:59336 worker1 # [ 127.535626] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:29Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 127.553566] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:29Z level=info event=scheduler_restarting worker2 # [ 127.557703] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:29Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 127.562194] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:29Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:35044 worker2 # [ 127.567622] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:29Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 127.572232] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:29Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:35044 worker2 # [ 127.577413] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:29Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 127.602744] nix-daemon[843]: reaped child process 1315, status = succeeded worker1 # [ 127.735274] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:29Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 127.741241] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 127.769065] nix-grpc-daemon[1262]: ts=2026-09-23T04:44:29Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker1 # [ 127.941797] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:29Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 127.947214] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 128.148074] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:29Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 128.153319] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 128.234959] nix-grpc-daemon[1262]: E0000 00:00:1790138669.753373 1308 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true worker2 # [ 128.242850] nix-grpc-daemon[1262]: E0000 00:00:1790138669.761093 1308 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker1 # [ 128.354495] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:29Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 128.359693] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 128.562518] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 128.566782] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 128.763223] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 128.768787] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 128.812214] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 128.821135] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 128.823727] systemd[1]: nix-grpc-daemon.service: Consumed 1.442s CPU time over 16.772s wall clock time, 39.4M memory peak, 46.5K incoming IP traffic, 82.9K outgoing IP traffic. worker2 # [ 128.839847] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker2 # [ 128.842158] systemd[1]: Closed Nix worker-protocol over gRPC. worker2 # [ 128.843794] systemd[1]: nix-grpc-daemon.socket: Consumed 878us CPU time over 17.450s wall clock time, 256K memory peak, 38.7K incoming IP traffic, 29.2K outgoing IP traffic. worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker1 # [ 128.969266] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 128.972147] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 129.173422] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 129.178898] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 129.380355] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 129.385701] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 129.518703] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:31Z level=info event=scheduler_take_over lb # [ 129.487369] niks3-server[1141]: 2026/09/23 04:44:31 INFO lead: acquired remote=192.168.1.3:41478 worker1 # [ 129.588743] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 129.594557] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 129.796190] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 129.801872] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1 # [ 130.002906] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 130.019256] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:31Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:56374 worker1 # [ 130.031953] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:31Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.11 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2 lb # [ 130.190745] envoy[891]: conn peer=[2001:db8:1::1]:46368 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9706 tx=4399 ms=5919 lb # [ 130.199381] envoy[891]: conn peer=[2001:db8:1::4]:50752 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=5513 tx=665 ms=15507 worker1 # [ 130.411872] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:42890 client # don't know how to build these paths: client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker1 # [ 130.424433] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 130.443471] nix-grpc-daemon[1224]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 46 ms (attempt 1/5) worker1 # [ 130.499533] nix-grpc-daemon[1224]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 52 ms (attempt 2/5) worker1 # [ 130.555061] nix-grpc-daemon[1224]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 331 ms (attempt 3/5) worker1 # [ 130.902054] nix-grpc-daemon[1224]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 389 ms (attempt 4/5) worker1 # [ 131.303563] nix-grpc-daemon[1224]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 131.311239] nix-grpc-daemon[1224]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 131.358165] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:32Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=0 worker1 # [ 131.365982] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:32Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 131.411106] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 131.433091] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:42890 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'... worker1 # [ 131.459483] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 131.481682] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted) worker1 # [ 131.524200] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:33Z level=info event=niks3_push_started pid=1301 worker1 # [ 131.753294] nix-grpc-daemon[1301]: time=2026-09-23T04:44:33.272Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 131.888138] nix-daemon[855]: accepted connection from pid 1314, user nix-grpc-daemon (trusted) worker1 # [ 131.912988] nix-daemon[855]: reaped child process 1317, status = succeeded lb # [ 131.893782] niks3-server[1141]: 2026/09/23 04:44:33 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 131.940040] nix-grpc-daemon[1301]: time=2026-09-23T04:44:33.458Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 131.941648] nix-grpc-daemon[1301]: time=2026-09-23T04:44:33.458Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)" lb # [ 131.958670] niks3-server[1141]: 2026/09/23 04:44:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign worker1 # [ 132.003308] nix-grpc-daemon[1301]: time=2026-09-23T04:44:33.521Z level=INFO msg="Uploading 1 narinfos" lb # [ 131.971180] niks3-server[1141]: 2026/09/23 04:44:33 INFO Signed narinfos id=43 count=1 lb # [ 131.986514] niks3-server[1141]: 2026/09/23 04:44:33 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls lb # [ 131.998289] niks3-server[1141]: 2026/09/23 04:44:33 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst lb # [ 132.024742] niks3-server[1141]: 2026/09/23 04:44:33 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete lb # [ 132.031489] niks3-server[1141]: 2026/09/23 04:44:33 INFO Completed upload id=43 worker1 # [ 132.063287] nix-grpc-daemon[1301]: time=2026-09-23T04:44:33.582Z level=INFO msg="Upload complete. (306ms)" lb # [ 132.037419] niks3-server[1141]: 2026/09/23 04:44:33 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo worker1 # [ 132.078677] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42890 duration_s=0 paths=1 nar_bytes_in=520 worker1 # [ 132.106024] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 132.116228] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted) worker1 # [ 132.123810] nix-daemon[855]: reaped child process 1299, status = succeeded client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051... worker1 # [ 132.142202] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted) client # farm-a> node-a: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-on-standby worker1 # [ 133.205085] nix-daemon[855]: reaped child process 1321, status = succeeded worker1 # [ 133.315604] nix-daemon[855]: accepted connection from pid 1337, user nix-grpc-daemon (trusted) worker1 # [ 133.335398] nix-daemon[855]: reaped child process 1340, status = succeeded lb # [ 133.307786] niks3-server[1141]: 2026/09/23 04:44:34 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 133.352042] nix-grpc-daemon[1301]: time=2026-09-23T04:44:34.869Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 133.353543] nix-grpc-daemon[1301]: time=2026-09-23T04:44:34.869Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)" lb # [ 133.342233] niks3-server[1141]: 2026/09/23 04:44:34 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst lb # [ 133.360586] niks3-server[1141]: 2026/09/23 04:44:34 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker1 # [ 133.404247] nix-grpc-daemon[1301]: time=2026-09-23T04:44:34.923Z level=INFO msg="Uploading 1 narinfos" lb # [ 133.371371] niks3-server[1141]: 2026/09/23 04:44:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign lb # [ 133.379770] niks3-server[1141]: 2026/09/23 04:44:34 INFO Signed narinfos id=44 count=1 lb # [ 133.384982] niks3-server[1141]: 2026/09/23 04:44:34 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls lb # [ 133.393214] niks3-server[1141]: 2026/09/23 04:44:34 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete worker1 # [ 133.431500] nix-grpc-daemon[1301]: time=2026-09-23T04:44:34.950Z level=INFO msg="Upload complete. (272ms)" lb # [ 133.403012] niks3-server[1141]: 2026/09/23 04:44:34 INFO Completed upload id=44 lb # [ 133.408555] niks3-server[1141]: 2026/09/23 04:44:34 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo worker1 # [ 133.456345] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42890 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=1 worker1 # [ 133.467871] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:34Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=0 worker1 # [ 133.471628] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:34Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 133.475598] nix-daemon[855]: reaped child process 1319, status = succeeded worker1 # [ 133.497266] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 133.513614] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 133.527531] nix-grpc-daemon[1301]: time=2026-09-23T04:44:35.046Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'... worker1 # [ 133.538460] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 133.554707] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted) worker1 # [ 133.706151] nix-daemon[855]: accepted connection from pid 1346, user nix-grpc-daemon (trusted) worker1 # [ 133.725642] nix-daemon[855]: reaped child process 1349, status = succeeded lb # [ 133.698206] niks3-server[1141]: 2026/09/23 04:44:35 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 133.739286] nix-grpc-daemon[1301]: time=2026-09-23T04:44:35.258Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 133.741656] nix-grpc-daemon[1301]: time=2026-09-23T04:44:35.260Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)" lb # [ 133.731112] niks3-server[1141]: 2026/09/23 04:44:35 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls worker1 # [ 133.772725] nix-grpc-daemon[1301]: time=2026-09-23T04:44:35.291Z level=INFO msg="Uploading 1 narinfos" lb # [ 133.742240] niks3-server[1141]: 2026/09/23 04:44:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign lb # [ 133.752595] niks3-server[1141]: 2026/09/23 04:44:35 INFO Signed narinfos id=45 count=1 lb # [ 133.765273] niks3-server[1141]: 2026/09/23 04:44:35 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst worker1 # [ 133.806697] nix-grpc-daemon[1301]: time=2026-09-23T04:44:35.326Z level=INFO msg="Upload complete. (235ms)" lb # [ 133.774588] niks3-server[1141]: 2026/09/23 04:44:35 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete lb # [ 133.779199] niks3-server[1141]: 2026/09/23 04:44:35 INFO Completed upload id=45 lb # [ 133.781919] niks3-server[1141]: 2026/09/23 04:44:35 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo worker1 # [ 133.816298] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42890 duration_s=0 paths=1 nar_bytes_in=640 worker1 # [ 133.833734] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 133.840600] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted) worker1 # [ 133.845860] nix-daemon[855]: reaped child process 1344, status = succeeded client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051... client # farm-b> node-a: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker1 # [ 133.859425] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-on-standby worker1 # [ 134.921098] nix-daemon[855]: reaped child process 1353, status = succeeded worker1 # [ 135.054365] nix-daemon[855]: accepted connection from pid 1369, user nix-grpc-daemon (trusted) worker1 # [ 135.073648] nix-daemon[855]: reaped child process 1372, status = succeeded lb # [ 135.042266] niks3-server[1141]: 2026/09/23 04:44:36 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 135.083672] nix-grpc-daemon[1301]: time=2026-09-23T04:44:36.601Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 135.086093] nix-grpc-daemon[1301]: time=2026-09-23T04:44:36.601Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)" lb # [ 135.083992] niks3-server[1141]: 2026/09/23 04:44:36 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker1 # [ 135.126564] nix-grpc-daemon[1301]: time=2026-09-23T04:44:36.645Z level=INFO msg="Uploading 1 narinfos" lb # [ 135.098624] niks3-server[1141]: 2026/09/23 04:44:36 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls lb # [ 135.106187] niks3-server[1141]: 2026/09/23 04:44:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign lb # [ 135.113059] niks3-server[1141]: 2026/09/23 04:44:36 INFO Signed narinfos id=46 count=1 lb # [ 135.116176] niks3-server[1141]: 2026/09/23 04:44:36 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst lb # [ 135.123444] niks3-server[1141]: 2026/09/23 04:44:36 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete lb # [ 135.127923] niks3-server[1141]: 2026/09/23 04:44:36 INFO Completed upload id=46 worker1 # [ 135.159866] nix-grpc-daemon[1301]: time=2026-09-23T04:44:36.679Z level=INFO msg="Upload complete. (289ms)" lb # [ 135.130965] niks3-server[1141]: 2026/09/23 04:44:36 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo worker1 # [ 135.180632] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42890 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=1 worker1 # [ 135.196227] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:36Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=0 worker1 # [ 135.200372] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:36Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 135.204902] nix-daemon[855]: reaped child process 1351, status = succeeded worker1 # [ 135.220749] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 135.234656] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 135.248593] nix-grpc-daemon[1301]: time=2026-09-23T04:44:36.768Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'... worker1 # [ 135.259741] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 135.274616] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted) worker1 # [ 135.416959] nix-daemon[855]: accepted connection from pid 1378, user nix-grpc-daemon (trusted) worker1 # [ 135.438086] nix-daemon[855]: reaped child process 1381, status = succeeded lb # [ 135.408820] niks3-server[1141]: 2026/09/23 04:44:36 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 135.451077] nix-grpc-daemon[1301]: time=2026-09-23T04:44:36.970Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 135.453148] nix-grpc-daemon[1301]: time=2026-09-23T04:44:36.970Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)" lb # [ 135.440209] niks3-server[1141]: 2026/09/23 04:44:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign worker1 # [ 135.481400] nix-grpc-daemon[1301]: time=2026-09-23T04:44:37.000Z level=INFO msg="Uploading 1 narinfos" lb # [ 135.456307] niks3-server[1141]: 2026/09/23 04:44:36 INFO Signed narinfos id=47 count=1 lb # [ 135.459342] niks3-server[1141]: 2026/09/23 04:44:36 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls lb # [ 135.469813] niks3-server[1141]: 2026/09/23 04:44:36 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst worker1 # [ 135.511502] nix-grpc-daemon[1301]: time=2026-09-23T04:44:37.030Z level=INFO msg="Upload complete. (223ms)" lb # [ 135.479860] niks3-server[1141]: 2026/09/23 04:44:37 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete lb # [ 135.485630] niks3-server[1141]: 2026/09/23 04:44:37 INFO Completed upload id=47 worker1 # [ 135.520261] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42890 duration_s=0 paths=1 nar_bytes_in=768 lb # [ 135.490972] niks3-server[1141]: 2026/09/23 04:44:37 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo worker1 # [ 135.535581] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 135.543044] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted) worker1 # [ 135.548121] nix-daemon[855]: reaped child process 1376, status = succeeded client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051... worker1 # [ 135.559157] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted) client # farm-top> node-a: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-on-standby worker1 # [ 136.856934] nix-daemon[855]: reaped child process 1385, status = succeeded worker1 # [ 136.926980] nix-daemon[855]: accepted connection from pid 1401, user nix-grpc-daemon (trusted) worker1 # [ 136.949326] nix-daemon[855]: reaped child process 1404, status = succeeded lb # [ 136.918016] niks3-server[1141]: 2026/09/23 04:44:38 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 136.960491] nix-grpc-daemon[1301]: time=2026-09-23T04:44:38.477Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 136.962357] nix-grpc-daemon[1301]: time=2026-09-23T04:44:38.477Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)" lb # [ 136.954809] niks3-server[1141]: 2026/09/23 04:44:38 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst lb # [ 136.970019] niks3-server[1141]: 2026/09/23 04:44:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign lb # [ 136.985425] niks3-server[1141]: 2026/09/23 04:44:38 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv lb # [ 136.994184] niks3-server[1141]: 2026/09/23 04:44:38 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls lb # [ 137.000841] niks3-server[1141]: 2026/09/23 04:44:38 INFO Signed narinfos id=48 count=1 worker1 # [ 137.033412] nix-grpc-daemon[1301]: time=2026-09-23T04:44:38.552Z level=INFO msg="Uploading 1 narinfos" lb # [ 137.009123] niks3-server[1141]: 2026/09/23 04:44:38 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete lb # [ 137.016608] niks3-server[1141]: 2026/09/23 04:44:38 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo lb # [ 137.029602] niks3-server[1141]: 2026/09/23 04:44:38 INFO Completed upload id=48 worker1 # [ 137.062315] nix-grpc-daemon[1301]: time=2026-09-23T04:44:38.581Z level=INFO msg="Upload complete. (291ms)" worker1 # [ 137.077151] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42890 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=1 worker1 # [ 137.095489] nix-daemon[855]: reaped child process 1383, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 7.05 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.18 seconds) worker2 # [ 137.398949] systemd[1]: Listening on Nix worker-protocol over gRPC. worker2 # [ 137.408508] systemd[1]: Starting scheduler-token.service... worker2 # [ 137.488862] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 137.493757] systemd[1]: Finished scheduler-token.service. worker2 # [ 137.495382] systemd[1]: scheduler-token.service: Consumed 44ms CPU time over 84ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 137.505405] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 137.634528] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:39Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 137.636961] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:39Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 137.641003] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:39Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 137.647957] nix-grpc-daemon[1469]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 137.663006] nix-grpc-daemon[1469]: E0000 00:00:1790138679.164206 1469 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 137.692146] nix-grpc-daemon[1469]: E0000 00:00:1790138679.166428 1469 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 137.696296] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:39Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 137.698539] nix-grpc-daemon[1469]: E0000 00:00:1790138679.176621 1509 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 137.701136] nix-grpc-daemon[1469]: E0000 00:00:1790138679.195790 1509 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 137.703382] nix-grpc-daemon[1469]: E0000 00:00:1790138679.195957 1509 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 137.709254] nix-grpc-daemon[1469]: E0000 00:00:1790138679.213856 1509 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 137.713731] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 137.714670] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 137.712194] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:39Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 137.717581] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:39Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:56374 worker1 # [ 137.722947] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:39Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.76 seconds) subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs client # [ 138.068917] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs. client # Running as unit: rs.service; invocation ID: 87f9f3c8d7bb460b980f2095e4a00487 client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.09 seconds) worker2 # [ 138.287999] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:39Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59580 worker2 # [ 138.305334] nix-daemon[843]: accepted connection from pid 1469, user nix-grpc-daemon (trusted) client # [ 138.302685] nix[1421]: don't know how to build these paths: client # [ 138.305122] nix[1421]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker1 # [ 138.326853] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:39Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=0 worker1 # [ 138.332412] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:39Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 138.373743] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 138.389696] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:42890 client # [ 138.396244] nix[1421]: copying 1 paths to 'lb:50051'... worker1 # [ 138.404352] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42890 client # [ 138.398552] nix[1421]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'... worker1 # [ 138.421401] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted) worker1 # [ 138.580273] nix-daemon[855]: accepted connection from pid 1417, user nix-grpc-daemon (trusted) worker1 # [ 138.599621] nix-daemon[855]: reaped child process 1420, status = succeeded lb # [ 138.571987] niks3-server[1141]: 2026/09/23 04:44:40 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 138.621196] nix-grpc-daemon[1301]: time=2026-09-23T04:44:40.140Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 138.622732] nix-grpc-daemon[1301]: time=2026-09-23T04:44:40.140Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)" lb # [ 138.614478] niks3-server[1141]: 2026/09/23 04:44:40 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls worker1 # [ 138.655146] nix-grpc-daemon[1301]: time=2026-09-23T04:44:40.174Z level=INFO msg="Uploading 1 narinfos" lb # [ 138.627563] niks3-server[1141]: 2026/09/23 04:44:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign lb # [ 138.637642] niks3-server[1141]: 2026/09/23 04:44:40 INFO Signed narinfos id=49 count=1 lb # [ 138.639853] niks3-server[1141]: 2026/09/23 04:44:40 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst worker1 # [ 138.680114] nix-grpc-daemon[1301]: time=2026-09-23T04:44:40.199Z level=INFO msg="Upload complete. (241ms)" lb # [ 138.649438] niks3-server[1141]: 2026/09/23 04:44:40 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete lb # [ 138.653518] niks3-server[1141]: 2026/09/23 04:44:40 INFO Completed upload id=49 lb # [ 138.657084] niks3-server[1141]: 2026/09/23 04:44:40 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo worker1 # [ 138.689349] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42890 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 138.703708] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42890 worker1 # [ 138.710350] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted) worker1 # [ 138.719975] nix-daemon[855]: reaped child process 1415, status = succeeded client # [ 138.717072] nix[1421]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051... worker1 # [ 138.727347] nix-daemon[855]: accepted connection from pid 1224, user nix-grpc-daemon (trusted) lb # [ 140.191175] envoy[891]: conn peer=[2001:db8:1::1]:55370 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9623 tx=4432 ms=6683 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true worker2 # [ 140.894751] systemd-journald[262]: Received client request to rotate journal, rotating. worker2 # [ 140.919314] systemd-journald[262]: Vacuuming done, freed 0B of archived journals from /var/log/journal/adab33dca4b240439adaf1a3266c8e0a. worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.09 seconds) worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service worker1 # [ 140.953907] systemd[1]: Stopping Nix worker-protocol over gRPC... worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds) client: must succeed: sleep 5.0 worker1 # [ 141.221925] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:42Z level=info event=draining client: (finished: must succeed: sleep 5.0, in 5.05 seconds) worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=' worker1 # [ 146.437786] nix-daemon[855]: reaped child process 1424, status = succeeded worker1 # [ 146.509664] nix-daemon[855]: accepted connection from pid 1465, user nix-grpc-daemon (trusted) worker1 # [ 146.530039] nix-daemon[855]: reaped child process 1468, status = succeeded lb # [ 146.496241] niks3-server[1141]: 2026/09/23 04:44:48 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 146.539358] nix-grpc-daemon[1301]: time=2026-09-23T04:44:48.058Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 146.543986] nix-grpc-daemon[1301]: time=2026-09-23T04:44:48.058Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)" lb # [ 146.543459] niks3-server[1141]: 2026/09/23 04:44:48 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst worker1 # [ 146.591002] nix-grpc-daemon[1301]: time=2026-09-23T04:44:48.110Z level=INFO msg="Uploading 1 narinfos" lb # [ 146.561513] niks3-server[1141]: 2026/09/23 04:44:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign lb # [ 146.572526] niks3-server[1141]: 2026/09/23 04:44:48 INFO Signed narinfos id=50 count=1 lb # [ 146.575072] niks3-server[1141]: 2026/09/23 04:44:48 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls worker1 # [ 146.619591] nix-grpc-daemon[1301]: time=2026-09-23T04:44:48.139Z level=INFO msg="Upload complete. (276ms)" lb # [ 146.587698] niks3-server[1141]: 2026/09/23 04:44:48 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv lb # [ 146.592179] niks3-server[1141]: 2026/09/23 04:44:48 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete lb # [ 146.596724] niks3-server[1141]: 2026/09/23 04:44:48 INFO Completed upload id=50 lb # [ 146.599985] niks3-server[1141]: 2026/09/23 04:44:48 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo worker1 # [ 146.633876] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42890 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=1 worker1 # [ 146.652957] nix-daemon[855]: reaped child process 1422, status = succeeded client # [ 146.663866] systemd[1]: rs.service: Deactivated successfully. client # [ 146.668113] systemd[1]: rs.service: Consumed 190ms CPU time over 8.598s wall clock time, 41.2M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic. worker1 # [ 147.223240] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:48Z level=info event=signal_exit lb # [ 147.488261] niks3-server[1141]: 2026/09/23 04:44:49 INFO lead: released remote=192.168.1.3:41478 lb # [ 147.614051] niks3-server[1141]: 2026/09/23 04:44:49 INFO lead: acquired remote=192.168.1.4:35882 worker2 # [ 147.663841] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:49Z level=info event=scheduler_take_over worker2 # [ 148.533830] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:50Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 148.527428] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:50Z level=info event=scheduler_restarting worker1 # [ 148.527893] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:56374 worker1 # [ 148.537140] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:50Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 148.542393] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:50Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 148.547035] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:56374 worker1 # [ 148.552558] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:50Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 148.577081] nix-daemon[855]: reaped child process 1277, status = succeeded worker2 # [ 148.736322] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:50Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 148.726337] nix-grpc-daemon[1224]: E0000 00:00:1790138690.245393 1270 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true worker1 # [ 148.754586] nix-grpc-daemon[1224]: ts=2026-09-23T04:44:50Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker1 # [ 148.956577] nix-grpc-daemon[1224]: E0000 00:00:1790138690.475692 1270 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true worker1 # [ 148.963122] nix-grpc-daemon[1224]: E0000 00:00:1790138690.476029 1270 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.24 seconds) client: waiting for success: ! systemctl is-active rs client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds) client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; } client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds) client: must fail: journalctl -u rs | grep -q 'reconnecting' client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker1 # [ 149.851023] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 149.856845] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 149.859414] systemd[1]: nix-grpc-daemon.service: Consumed 1.744s CPU time over 32.806s wall clock time, 39.1M memory peak, 48.3K incoming IP traffic, 90.7K outgoing IP traffic. worker2 # [ 149.887736] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:51Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 149.892279] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:51Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:33004 worker1 # [ 149.882826] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 149.897633] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:51Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 150.024719] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:51Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 150.026665] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:51Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 150.030527] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:51Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 150.036061] nix-grpc-daemon[1475]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 150.037518] nix-grpc-daemon[1475]: E0000 00:00:1790138691.554589 1475 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 150.074464] nix-grpc-daemon[1475]: E0000 00:00:1790138691.561014 1475 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 150.081461] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:51Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 150.083639] nix-grpc-daemon[1475]: E0000 00:00:1790138691.567669 1515 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 150.086378] nix-grpc-daemon[1475]: E0000 00:00:1790138691.583112 1515 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 150.091738] nix-grpc-daemon[1475]: E0000 00:00:1790138691.583269 1515 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 150.096186] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 150.097156] nix-grpc-daemon[1475]: E0000 00:00:1790138691.600069 1515 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 150.099412] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:51Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 150.109028] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:51Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:33004 worker2 # [ 150.115214] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:51Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb # [ 150.189631] envoy[891]: conn peer=[2001:db8:1::1]:33906 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3005 tx=1516 ms=8378 lb # [ 150.194551] envoy[891]: conn peer=[2001:db8:1::3]:50178 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2987 tx=1005 ms=31466 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.92 seconds) subtest: scheduler restart mid-build: build finishes, no double build client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid client # [ 150.990481] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid. client # Running as unit: mid.service; invocation ID: 73a181792c2e4aa7ba27687a6bd9027d client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.09 seconds) worker2 # [ 151.205704] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:52Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59580 client # [ 151.193988] nix[1457]: don't know how to build these paths: client # [ 151.194933] nix[1457]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker2 # [ 151.225310] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:52Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker2 # [ 151.232341] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:52Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 151.273024] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59580 worker2 # [ 151.288615] nix-grpc-daemon[1469]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 4 ms (attempt 1/5) worker2 # [ 151.301531] nix-grpc-daemon[1469]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 43 ms (attempt 2/5) worker2 # [ 151.353277] nix-grpc-daemon[1469]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 53 ms (attempt 3/5) worker2 # [ 151.407968] nix-grpc-daemon[1469]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 750 ms (attempt 4/5) worker2 # [ 152.160372] nix-grpc-daemon[1469]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 152.164584] nix-grpc-daemon[1469]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 152.184760] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59580 client # [ 152.177868] nix[1457]: copying 1 paths to 'lb:50051'... client # [ 152.182398] nix[1457]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'... worker2 # [ 152.203070] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59580 worker2 # [ 152.218130] nix-daemon[843]: accepted connection from pid 1469, user nix-grpc-daemon (trusted) worker2 # [ 152.256906] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:53Z level=info event=niks3_push_started pid=1598 worker2 # [ 152.416659] nix-grpc-daemon[1598]: time=2026-09-23T04:44:53.932Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 152.523100] nix-daemon[843]: accepted connection from pid 1615, user nix-grpc-daemon (trusted) worker2 # [ 152.541949] nix-daemon[843]: reaped child process 1618, status = succeeded lb # [ 152.502751] niks3-server[1141]: 2026/09/23 04:44:54 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 152.558667] nix-grpc-daemon[1598]: time=2026-09-23T04:44:54.076Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 152.562462] nix-grpc-daemon[1598]: time=2026-09-23T04:44:54.076Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)" lb # [ 152.546262] niks3-server[1141]: 2026/09/23 04:44:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign lb # [ 152.560344] niks3-server[1141]: 2026/09/23 04:44:54 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls lb # [ 152.577244] niks3-server[1141]: 2026/09/23 04:44:54 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst lb # [ 152.586806] niks3-server[1141]: 2026/09/23 04:44:54 INFO Signed narinfos id=51 count=1 worker2 # [ 152.631523] nix-grpc-daemon[1598]: time=2026-09-23T04:44:54.148Z level=INFO msg="Uploading 1 narinfos" lb # [ 152.598394] niks3-server[1141]: 2026/09/23 04:44:54 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete lb # [ 152.611516] niks3-server[1141]: 2026/09/23 04:44:54 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo lb # [ 152.622216] niks3-server[1141]: 2026/09/23 04:44:54 INFO Completed upload id=51 worker2 # [ 152.667284] nix-grpc-daemon[1598]: time=2026-09-23T04:44:54.185Z level=INFO msg="Upload complete. (252ms)" worker2 # [ 152.680311] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59580 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 152.698686] nix-grpc-daemon[1469]: ts=2026-09-23T04:44:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59580 worker2 # [ 152.703024] nix-daemon[843]: accepted connection from pid 1469, user nix-grpc-daemon (trusted) worker2 # [ 152.710489] nix-daemon[843]: reaped child process 1596, status = succeeded worker2 # [ 152.722441] nix-daemon[843]: accepted connection from pid 1469, user nix-grpc-daemon (trusted) client # [ 152.702528] nix[1457]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service worker2 # [ 154.994346] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1469 (nix-grpc-daemon) on client request. worker2 # [ 154.995928] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request. lb # [ 154.966706] niks3-server[1141]: 2026/09/23 04:44:56 INFO lead: released remote=192.168.1.4:35882 client # [ 154.999392] nix[1457]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again worker1 # [ 155.006475] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:56Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination" worker2 # [ 155.026790] systemd[1]: Starting scheduler-token.service... worker2 # [ 155.027675] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL client # [ 155.005774] nix[1457]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting worker2 # [ 155.038361] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'. worker2 # [ 155.039665] systemd[1]: nix-grpc-daemon.service: Consumed 503ms CPU time over 17.519s wall clock time, 34.4M memory peak, 15.2K incoming IP traffic, 23.8K outgoing IP traffic. lb # [ 155.017203] niks3-server[1141]: 2026/09/23 04:44:56 INFO lead: acquired remote=192.168.1.3:35522 worker1 # [ 155.053782] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:56Z level=info event=scheduler_take_over worker2 # [ 155.072404] nix-daemon[843]: reaped child process 1620, status = succeeded worker2 # [ 155.075260] nix-daemon[843]: reaped child process 1529, status = succeeded worker2 # [ 155.080464] nix-daemon[1622]: unexpected Nix daemon error: error: interrupted by the user worker2 # [ 155.090996] nix-daemon[843]: reaped child process 1622, status = failed with exit code 1 worker2 # [ 155.110875] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 155.115031] systemd[1]: Finished scheduler-token.service. worker2 # [ 155.115834] systemd[1]: scheduler-token.service: Consumed 24ms CPU time over 88ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 155.119654] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1. worker2 # [ 155.124272] systemd[1]: Starting scheduler-token.service... worker2 # [ 155.165743] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 155.167994] systemd[1]: Finished scheduler-token.service. worker2 # [ 155.170676] systemd[1]: scheduler-token.service: Consumed 22ms CPU time over 48ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 155.179453] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 155.206573] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:56Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 155.273768] nix-grpc-daemon[1665]: ts=2026-09-23T04:44:56Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 155.276286] nix-grpc-daemon[1665]: ts=2026-09-23T04:44:56Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 155.279987] nix-grpc-daemon[1665]: ts=2026-09-23T04:44:56Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 155.286432] nix-grpc-daemon[1665]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 155.287887] nix-grpc-daemon[1665]: E0000 00:00:1790138696.802994 1665 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 155.294533] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:56Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.36 seconds) client: must succeed: sleep 5.0 worker2 # [ 155.332115] nix-grpc-daemon[1665]: E0000 00:00:1790138696.804935 1665 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 155.336478] nix-grpc-daemon[1665]: ts=2026-09-23T04:44:56Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 155.338632] nix-grpc-daemon[1665]: E0000 00:00:1790138696.820647 1705 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 155.341254] nix-grpc-daemon[1665]: E0000 00:00:1790138696.837339 1705 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 155.346215] nix-grpc-daemon[1665]: E0000 00:00:1790138696.837492 1705 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 155.350947] nix-grpc-daemon[1665]: E0000 00:00:1790138696.854288 1705 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 155.353904] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 155.354726] nix-grpc-daemon[1665]: ts=2026-09-23T04:44:56Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 155.355994] nix-grpc-daemon[1665]: ts=2026-09-23T04:44:56Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker1 # [ 155.495381] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 155.500836] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" client # [ 155.503798] nix[1457]: scheduler at lb:50051: scheduler passive, another node serves, reconnecting worker2 # [ 155.555195] nix-grpc-daemon[1665]: ts=2026-09-23T04:44:57Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 155.561931] nix-grpc-daemon[1665]: ts=2026-09-23T04:44:57Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker1 # [ 155.701578] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 155.706909] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker2 # [ 155.763081] nix-grpc-daemon[1665]: ts=2026-09-23T04:44:57Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 155.769755] nix-grpc-daemon[1665]: ts=2026-09-23T04:44:57Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker1 # [ 155.907505] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 155.918129] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker2 # [ 155.970935] nix-grpc-daemon[1665]: ts=2026-09-23T04:44:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 155.980231] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 155.984677] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:35602 worker1 # [ 155.989994] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 156.120387] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 156.125557] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker1 # [ 156.326429] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 156.332667] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:35602 worker1 # [ 156.337551] nix-grpc-daemon[1475]: ts=2026-09-23T04:44:57Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 client # [ 156.510881] nix[1457]: scheduler at lb:50051: scheduler passive, another node serves, reconnecting worker1 # [ 158.532515] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:00Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker1 # [ 158.532838] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:00Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 158.595557] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41730 worker2 # [ 158.610619] nix-daemon[843]: accepted connection from pid 1665, user nix-grpc-daemon (trusted) worker2 # [ 158.633212] nix-daemon[843]: accepted connection from pid 1665, user nix-grpc-daemon (trusted) client # [ 158.629482] nix[1457]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... worker2 # [ 158.652259] nix-daemon[843]: accepted connection from pid 1665, user nix-grpc-daemon (trusted) lb # [ 160.190201] envoy[891]: conn peer=[2001:db8:1::4]:36054 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=1935 tx=328 ms=17317 client: (finished: must succeed: sleep 5.0, in 5.05 seconds) client: waiting for success: ! systemctl is-active mid worker2 # [ 160.680385] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:02Z level=info event=niks3_push_started pid=1747 worker2 # [ 160.690542] nix-daemon[843]: reaped child process 1722, status = succeeded worker2 # [ 160.804105] nix-grpc-daemon[1747]: time=2026-09-23T04:45:02.322Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 160.910722] nix-daemon[843]: accepted connection from pid 1760, user nix-grpc-daemon (trusted) worker2 # [ 160.930891] nix-daemon[843]: reaped child process 1763, status = succeeded lb # [ 160.893211] niks3-server[1141]: 2026/09/23 04:45:02 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 160.951418] nix-grpc-daemon[1747]: time=2026-09-23T04:45:02.468Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 160.954981] nix-grpc-daemon[1747]: time=2026-09-23T04:45:02.468Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)" lb # [ 160.950531] niks3-server[1141]: 2026/09/23 04:45:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign worker2 # [ 161.008905] nix-grpc-daemon[1747]: time=2026-09-23T04:45:02.526Z level=INFO msg="Uploading 1 narinfos" lb # [ 160.964574] niks3-server[1141]: 2026/09/23 04:45:02 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv lb # [ 160.977523] niks3-server[1141]: 2026/09/23 04:45:02 INFO Signed narinfos id=52 count=1 lb # [ 160.979357] niks3-server[1141]: 2026/09/23 04:45:02 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls lb # [ 160.995231] niks3-server[1141]: 2026/09/23 04:45:02 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete worker2 # [ 161.045046] nix-grpc-daemon[1747]: time=2026-09-23T04:45:02.563Z level=INFO msg="Upload complete. (237ms)" lb # [ 161.004278] niks3-server[1141]: 2026/09/23 04:45:02 INFO Completed upload id=52 lb # [ 161.007003] niks3-server[1141]: 2026/09/23 04:45:02 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst lb # [ 161.011611] niks3-server[1141]: 2026/09/23 04:45:02 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo worker2 # [ 161.055303] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41730 duration_s=2 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=1 worker2 # [ 161.074821] nix-daemon[843]: reaped child process 1720, status = succeeded client # [ 161.074531] systemd[1]: mid.service: Deactivated successfully. client # [ 161.078737] systemd[1]: mid.service: Consumed 202ms CPU time over 10.085s wall clock time, 41.4M memory peak, 9.4K incoming IP traffic, 11.3K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active mid, in 1.12 seconds) client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; } client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.05 seconds) (finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.84 seconds) subtest: interrupting the client stops the build on the worker client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr client # [ 161.809506] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr. client # Running as unit: intr.service; invocation ID: f8e7fca39b434e7dbf199bd5e4c1a94a client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.05 seconds) worker1 # [ 162.010054] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 162.024432] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) client # [ 162.036450] nix[1493]: don't know how to build these paths: client # [ 162.038346] nix[1493]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker1 # [ 162.059226] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:03Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=0 worker1 # [ 162.065987] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:03Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 162.105189] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 162.121369] nix-grpc-daemon[1475]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 31 ms (attempt 1/5) worker1 # [ 162.162105] nix-grpc-daemon[1475]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 61 ms (attempt 2/5) worker1 # [ 162.225080] nix-grpc-daemon[1475]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 266 ms (attempt 3/5) worker1 # [ 162.493374] nix-grpc-daemon[1475]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 663 ms (attempt 4/5) worker1 # [ 163.160658] nix-grpc-daemon[1475]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 163.165179] nix-grpc-daemon[1475]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 163.185213] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 client # [ 163.193408] nix[1493]: copying 1 paths to 'lb:50051'... worker1 # [ 163.201264] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 client # [ 163.196744] nix[1493]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'... worker1 # [ 163.220649] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 163.249389] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:04Z level=info event=niks3_push_started pid=1576 worker1 # [ 163.395684] nix-grpc-daemon[1576]: time=2026-09-23T04:45:04.914Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 163.502828] nix-daemon[855]: accepted connection from pid 1589, user nix-grpc-daemon (trusted) worker1 # [ 163.523150] nix-daemon[855]: reaped child process 1592, status = succeeded lb # [ 163.492693] niks3-server[1141]: 2026/09/23 04:45:05 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 163.536074] nix-grpc-daemon[1576]: time=2026-09-23T04:45:05.054Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 163.537595] nix-grpc-daemon[1576]: time=2026-09-23T04:45:05.054Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)" lb # [ 163.532225] niks3-server[1141]: 2026/09/23 04:45:05 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst worker1 # [ 163.576042] nix-grpc-daemon[1576]: time=2026-09-23T04:45:05.094Z level=INFO msg="Uploading 1 narinfos" lb # [ 163.547511] niks3-server[1141]: 2026/09/23 04:45:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign lb # [ 163.555903] niks3-server[1141]: 2026/09/23 04:45:05 INFO Signed narinfos id=53 count=1 worker1 # [ 163.597929] nix-grpc-daemon[1576]: time=2026-09-23T04:45:05.117Z level=INFO msg="Upload complete. (199ms)" lb # [ 163.565462] niks3-server[1141]: 2026/09/23 04:45:05 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete worker1 # [ 163.604288] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=424 lb # [ 163.575220] niks3-server[1141]: 2026/09/23 04:45:05 INFO Completed upload id=53 lb # [ 163.578013] niks3-server[1141]: 2026/09/23 04:45:05 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls worker1 # [ 163.613316] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 163.622462] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 163.626841] nix-daemon[855]: reaped child process 1574, status = succeeded worker1 # [ 163.635673] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) client # [ 163.628340] nix[1493]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051... client: must succeed: systemctl kill -s INT intr worker1 # [ 165.464854] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:06Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv client # [ 165.453734] systemd[1]: intr.service: Sent signal SIGINT to main process 1493 (nix) on client request. client # [ 165.463508] nix[1493]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client: (finished: must succeed: systemctl kill -s INT intr, in 0.07 seconds) client # [ 165.480446] nix[1493]: [mutex.cc : 451] RAW: Unlock 0xebfeff9fc788 @ 0xec03040b59a4 0xec030518d40c 0xec03245f57bc 0xec03245f7414 0xec03245ee1d4 0xec0323634f7c 0xec03233c2d80 client # [ 165.492787] systemd-coredump[1507]: Process 1493 (nix) of user 0 terminated abnormally with signal 11/SEGV, processing... client # [ 165.509136] systemd[1]: Started Process Core Dump (PID 1507/UID 0). worker1 # [ 165.662025] nix-daemon[855]: reaped child process 1594, status = succeeded worker1 # [ 165.663866] nix-daemon[1596]: unexpected Nix daemon error: error: interrupted by the user worker1 # [ 165.672938] nix-daemon[855]: reaped child process 1596, status = failed with exit code 1 (finished: subtest: interrupting the client stops the build on the worker, in 4.98 seconds) subtest: a deploy mid-build drains: the build finishes, the new generation takes over client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw client # [ 166.794354] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw. client # Running as unit: sw.service; invocation ID: 0c9f581f3a7d423788777dcd1c159ea9 client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.06 seconds) worker1 # [ 167.155844] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:32848 client # [ 167.151101] nix[1514]: don't know how to build these paths: client # [ 167.153405] nix[1514]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker1 # [ 167.166085] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:08Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=0 worker1 # [ 167.169629] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:08Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 167.226734] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41730 worker2 # [ 167.242781] nix-grpc-daemon[1665]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 0 ms (attempt 1/5) worker2 # [ 167.252220] nix-grpc-daemon[1665]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 134 ms (attempt 2/5) client # [ 167.343504] systemd-coredump[1508]: Cannot store coredump of 1493 (nix): No space left on device worker2 # [ 167.387987] nix-grpc-daemon[1665]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 77 ms (attempt 3/5) worker2 # [ 167.466903] nix-grpc-daemon[1665]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 157 ms (attempt 4/5) client # [ 167.468527] systemd-coredump[1508]: Process 1493 (nix) of user 0 terminated abnormally without generating a coredump. client # [ 167.480860] systemd[1]: systemd-coredump@1-2-1507_1508-0.service: Deactivated successfully. client # [ 167.483951] systemd[1]: systemd-coredump@1-2-1507_1508-0.service: Consumed 1.391s CPU time over 1.973s wall clock time, 968.6M memory peak, 926.1M written to disk. client # [ 167.494520] systemd[1]: intr.service: Main process exited, code=dumped, status=11/SEGV client # [ 167.497382] systemd[1]: intr.service: Failed with result 'core-dump'. client # [ 167.498574] systemd[1]: intr.service: Consumed 362ms CPU time over 5.686s wall clock time, 41.5M memory peak, 4K written to disk, 6.6K incoming IP traffic, 8.2K outgoing IP traffic. worker2 # [ 167.634060] nix-grpc-daemon[1665]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 167.640380] nix-grpc-daemon[1665]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 167.659434] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41730 client # [ 167.656173] nix[1514]: copying 1 paths to 'lb:50051'... worker2 # [ 167.675725] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41730 client # [ 167.658806] nix[1514]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'... worker2 # [ 167.690766] nix-daemon[843]: accepted connection from pid 1665, user nix-grpc-daemon (trusted) worker2 # [ 167.849499] nix-daemon[843]: accepted connection from pid 1799, user nix-grpc-daemon (trusted) worker2 # [ 167.868553] nix-daemon[843]: reaped child process 1802, status = succeeded lb # [ 167.831192] niks3-server[1141]: 2026/09/23 04:45:09 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 167.889562] nix-grpc-daemon[1747]: time=2026-09-23T04:45:09.407Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 167.891201] nix-grpc-daemon[1747]: time=2026-09-23T04:45:09.407Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)" lb # [ 167.863736] niks3-server[1141]: 2026/09/23 04:45:09 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls worker2 # [ 167.922496] nix-grpc-daemon[1747]: time=2026-09-23T04:45:09.440Z level=INFO msg="Uploading 1 narinfos" lb # [ 167.877944] niks3-server[1141]: 2026/09/23 04:45:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign lb # [ 167.889193] niks3-server[1141]: 2026/09/23 04:45:09 INFO Signed narinfos id=54 count=1 lb # [ 167.895585] niks3-server[1141]: 2026/09/23 04:45:09 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst lb # [ 167.908294] niks3-server[1141]: 2026/09/23 04:45:09 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete lb # [ 167.919289] niks3-server[1141]: 2026/09/23 04:45:09 INFO Completed upload id=54 worker2 # [ 167.963627] nix-grpc-daemon[1747]: time=2026-09-23T04:45:09.482Z level=INFO msg="Upload complete. (256ms)" lb # [ 167.922162] niks3-server[1141]: 2026/09/23 04:45:09 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo worker2 # [ 167.974295] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41730 duration_s=0 paths=1 nar_bytes_in=416 worker2 # [ 167.988605] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41730 worker2 # [ 168.000512] nix-daemon[843]: accepted connection from pid 1665, user nix-grpc-daemon (trusted) worker2 # [ 168.005697] nix-daemon[843]: reaped child process 1797, status = succeeded worker2 # [ 168.018623] nix-daemon[843]: accepted connection from pid 1665, user nix-grpc-daemon (trusted) client # [ 167.997402] nix[1514]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051... lb # [ 170.191125] envoy[891]: conn peer=[2001:db8:1::1]:59954 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4314 tx=2076 ms=9867 worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2 worker2 # Checking switch inhibitors... done lb # [ 170.196808] envoy[891]: conn peer=[2001:db8:1::1]:59384 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554536:system_library:OPENSSL_internal:Connection_reset_by_peer:TLS_error_end" rx=2987 tx=887 ms=5505 worker2 # [ 170.931485] nixos[1842]: switching to system configuration /nix/store/8la61jwrdhbab16camb5i9ydwcpwnkl9-nixos-system-worker2-test worker2 # activating the configuration... worker2 # [ 170.938023] systemd[1]: Stopped target Local File Systems. worker2 # [ 170.938824] systemd[1]: Stopped target Remote File Systems. worker2 # setting up /etc... worker2 # [ 171.480706] systemd[1]: Reload requested from client PID 1842 ('.switch-to-conf') (unit backdoor.service)... worker2 # [ 171.482226] systemd[1]: Reloading... worker2 # [ 176.363583] systemd[1]: Reloading finished in 4871 ms. worker2 # restarting sysinit-reactivation.target worker2 # [ 176.492767] systemd[1]: Starting Re-setup tmpfiles on a system that is already running.... worker2 # [ 176.619525] systemd[1]: Finished Re-setup tmpfiles on a system that is already running.. worker2 # [ 176.621403] systemd[1]: Reached target Reactivate sysinit units. worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service worker2 # [ 176.628403] systemd[1]: Reloading D-Bus System Message Bus... worker2 # [ 176.631202] dbus-broker-launch[485]: Caught SIGHUP, trigger reload. worker2 # [ 176.633786] systemd[1]: Reloading Nix worker-protocol over gRPC... worker2 # [ 176.647938] dbus-broker-launch[485]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 176.649988] dbus-broker-launch[485]: NSS returned no entry for 'systemd-timesync' worker2 # [ 176.651013] dbus-broker-launch[485]: Invalid user-name in /nix/store/ng2ydfkv03gvmdws4f4mjf9wj6qj3kq6-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker2 # [ 176.675950] systemd[1]: Reloaded Nix worker-protocol over gRPC. worker2 # [ 176.693484] systemd[1]: Reloaded D-Bus System Message Bus. worker2 # [ 176.696496] systemd[1]: Reached target Remote File Systems. worker2 # [ 176.702249] systemd[1]: Reached target Local File Systems. worker2 # [ 176.703075] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker2 # [ 176.712425] systemd[1]: Starting Create SUID/SGID Wrappers... worker2 # [ 176.713300] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker2 # [ 176.714464] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker2 # [ 176.728132] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 176.729971] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 176.731204] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 176.736385] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 176.737655] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 177.045882] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 177.046952] systemd[1]: Finished Create SUID/SGID Wrappers. worker2 # [ 177.052546] systemd[1]: Starting linger-users.service... worker2 # [ 177.093279] systemd[1]: linger-users.service: Deactivated successfully. worker2 # [ 177.094200] systemd[1]: Finished linger-users.service. worker2 # [ 177.299313] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:18Z level=info event=draining worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service worker2 # [ 177.353945] nixos[1842]: finished switching to system configuration /nix/store/8la61jwrdhbab16camb5i9ydwcpwnkl9-nixos-system-worker2-test worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 6.87 seconds) worker2: must succeed: pgrep -f 'read -t [9]0 x' worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2 worker2 # [ 177.623461] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:41730 client # don't know how to build these paths: client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker1 # [ 177.619178] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 177.646428] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:19Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=0 worker1 # [ 177.653662] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:19Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 177.692682] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 177.708431] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'... worker1 # [ 177.724113] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 177.738899] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 177.893136] nix-daemon[855]: accepted connection from pid 1655, user nix-grpc-daemon (trusted) worker1 # [ 177.914966] nix-daemon[855]: reaped child process 1658, status = succeeded lb # [ 177.881272] niks3-server[1141]: 2026/09/23 04:45:19 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 177.925639] nix-grpc-daemon[1576]: time=2026-09-23T04:45:19.444Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 177.927123] nix-grpc-daemon[1576]: time=2026-09-23T04:45:19.444Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)" lb # [ 177.912872] niks3-server[1141]: 2026/09/23 04:45:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign worker1 # [ 177.954704] nix-grpc-daemon[1576]: time=2026-09-23T04:45:19.473Z level=INFO msg="Uploading 1 narinfos" lb # [ 177.929244] niks3-server[1141]: 2026/09/23 04:45:19 INFO Signed narinfos id=55 count=1 lb # [ 177.937878] niks3-server[1141]: 2026/09/23 04:45:19 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls lb # [ 177.945582] niks3-server[1141]: 2026/09/23 04:45:19 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst lb # [ 177.958437] niks3-server[1141]: 2026/09/23 04:45:19 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete lb # [ 177.964339] niks3-server[1141]: 2026/09/23 04:45:19 INFO Completed upload id=55 worker1 # [ 177.995365] nix-grpc-daemon[1576]: time=2026-09-23T04:45:19.514Z level=INFO msg="Upload complete. (240ms)" lb # [ 177.966518] niks3-server[1141]: 2026/09/23 04:45:19 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo worker1 # [ 178.004354] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=528 worker1 # [ 178.022827] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 178.030242] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 178.035839] nix-daemon[855]: reaped child process 1653, status = succeeded client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051... worker1 # [ 178.049843] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) client # farm-a> node-a: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-during-drain worker1 # [ 179.120241] nix-daemon[855]: reaped child process 1662, status = succeeded worker1 # [ 179.236422] nix-daemon[855]: accepted connection from pid 1678, user nix-grpc-daemon (trusted) worker1 # [ 179.255396] nix-daemon[855]: reaped child process 1681, status = succeeded lb # [ 179.227609] niks3-server[1141]: 2026/09/23 04:45:20 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 179.276592] nix-grpc-daemon[1576]: time=2026-09-23T04:45:20.793Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 179.278755] nix-grpc-daemon[1576]: time=2026-09-23T04:45:20.793Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)" lb # [ 179.259334] niks3-server[1141]: 2026/09/23 04:45:20 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv lb # [ 179.276729] niks3-server[1141]: 2026/09/23 04:45:20 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst lb # [ 179.288881] niks3-server[1141]: 2026/09/23 04:45:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign worker1 # [ 179.321858] nix-grpc-daemon[1576]: time=2026-09-23T04:45:20.841Z level=INFO msg="Uploading 1 narinfos" lb # [ 179.292912] niks3-server[1141]: 2026/09/23 04:45:20 INFO Signed narinfos id=56 count=1 lb # [ 179.297291] niks3-server[1141]: 2026/09/23 04:45:20 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls lb # [ 179.303913] niks3-server[1141]: 2026/09/23 04:45:20 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete lb # [ 179.307684] niks3-server[1141]: 2026/09/23 04:45:20 INFO Completed upload id=56 worker1 # [ 179.339839] nix-grpc-daemon[1576]: time=2026-09-23T04:45:20.859Z level=INFO msg="Upload complete. (275ms)" lb # [ 179.311742] niks3-server[1141]: 2026/09/23 04:45:20 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo worker1 # [ 179.357073] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=1 worker1 # [ 179.372298] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:20Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=0 worker1 # [ 179.376837] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:20Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 179.380601] nix-daemon[855]: reaped child process 1660, status = succeeded worker1 # [ 179.401037] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 179.414289] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 179.428625] nix-grpc-daemon[1576]: time=2026-09-23T04:45:20.948Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'... worker1 # [ 179.439324] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 179.452847] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 179.601044] nix-daemon[855]: accepted connection from pid 1687, user nix-grpc-daemon (trusted) worker1 # [ 179.618146] nix-daemon[855]: reaped child process 1690, status = succeeded lb # [ 179.592479] niks3-server[1141]: 2026/09/23 04:45:21 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 179.637300] nix-grpc-daemon[1576]: time=2026-09-23T04:45:21.156Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 179.638929] nix-grpc-daemon[1576]: time=2026-09-23T04:45:21.156Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)" lb # [ 179.627284] niks3-server[1141]: 2026/09/23 04:45:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign worker1 # [ 179.667204] nix-grpc-daemon[1576]: time=2026-09-23T04:45:21.186Z level=INFO msg="Uploading 1 narinfos" lb # [ 179.641312] niks3-server[1141]: 2026/09/23 04:45:21 INFO Signed narinfos id=57 count=1 lb # [ 179.643936] niks3-server[1141]: 2026/09/23 04:45:21 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst worker1 # [ 179.695584] nix-grpc-daemon[1576]: time=2026-09-23T04:45:21.215Z level=INFO msg="Upload complete. (227ms)" lb # [ 179.663628] niks3-server[1141]: 2026/09/23 04:45:21 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls lb # [ 179.673515] niks3-server[1141]: 2026/09/23 04:45:21 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete worker1 # [ 179.704283] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=640 lb # [ 179.678323] niks3-server[1141]: 2026/09/23 04:45:21 INFO Completed upload id=57 lb # [ 179.680812] niks3-server[1141]: 2026/09/23 04:45:21 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo worker1 # [ 179.719860] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 179.728318] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 179.732928] nix-daemon[855]: reaped child process 1685, status = succeeded client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051... client # farm-b> node-a: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker1 # [ 179.747149] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-during-drain worker1 # [ 180.610820] nix-daemon[855]: reaped child process 1694, status = succeeded worker1 # [ 180.742230] nix-daemon[855]: accepted connection from pid 1710, user nix-grpc-daemon (trusted) worker1 # [ 180.758290] nix-daemon[855]: reaped child process 1713, status = succeeded lb # [ 180.734204] niks3-server[1141]: 2026/09/23 04:45:22 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 180.777454] nix-grpc-daemon[1576]: time=2026-09-23T04:45:22.295Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 180.779122] nix-grpc-daemon[1576]: time=2026-09-23T04:45:22.295Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)" lb # [ 180.772130] niks3-server[1141]: 2026/09/23 04:45:22 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst worker1 # [ 180.818593] nix-grpc-daemon[1576]: time=2026-09-23T04:45:22.337Z level=INFO msg="Uploading 1 narinfos" lb # [ 180.785536] niks3-server[1141]: 2026/09/23 04:45:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign lb # [ 180.798143] niks3-server[1141]: 2026/09/23 04:45:22 INFO Signed narinfos id=58 count=1 lb # [ 180.804482] niks3-server[1141]: 2026/09/23 04:45:22 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv lb # [ 180.813424] niks3-server[1141]: 2026/09/23 04:45:22 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls lb # [ 180.826107] niks3-server[1141]: 2026/09/23 04:45:22 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete worker1 # [ 180.858990] nix-grpc-daemon[1576]: time=2026-09-23T04:45:22.377Z level=INFO msg="Upload complete. (301ms)" lb # [ 180.831041] niks3-server[1141]: 2026/09/23 04:45:22 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo lb # [ 180.836181] niks3-server[1141]: 2026/09/23 04:45:22 INFO Completed upload id=58 worker1 # [ 180.876182] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=1 worker1 # [ 180.894907] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:22Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=0 worker1 # [ 180.901620] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:22Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 180.906666] nix-daemon[855]: reaped child process 1692, status = succeeded worker1 # [ 180.940477] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 180.955067] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 180.969963] nix-grpc-daemon[1576]: time=2026-09-23T04:45:22.489Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'... worker1 # [ 180.979985] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 180.993122] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 181.144371] nix-daemon[855]: accepted connection from pid 1719, user nix-grpc-daemon (trusted) worker1 # [ 181.161245] nix-daemon[855]: reaped child process 1722, status = succeeded lb # [ 181.133892] niks3-server[1141]: 2026/09/23 04:45:22 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 181.175085] nix-grpc-daemon[1576]: time=2026-09-23T04:45:22.694Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 181.176864] nix-grpc-daemon[1576]: time=2026-09-23T04:45:22.694Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)" lb # [ 181.164240] niks3-server[1141]: 2026/09/23 04:45:22 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst worker1 # [ 181.206153] nix-grpc-daemon[1576]: time=2026-09-23T04:45:22.725Z level=INFO msg="Uploading 1 narinfos" lb # [ 181.179608] niks3-server[1141]: 2026/09/23 04:45:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign lb # [ 181.189655] niks3-server[1141]: 2026/09/23 04:45:22 INFO Signed narinfos id=59 count=1 lb # [ 181.198226] niks3-server[1141]: 2026/09/23 04:45:22 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls worker1 # [ 181.236126] nix-grpc-daemon[1576]: time=2026-09-23T04:45:22.755Z level=INFO msg="Upload complete. (226ms)" lb # [ 181.204052] niks3-server[1141]: 2026/09/23 04:45:22 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete lb # [ 181.207946] niks3-server[1141]: 2026/09/23 04:45:22 INFO Completed upload id=59 lb # [ 181.211296] niks3-server[1141]: 2026/09/23 04:45:22 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo worker1 # [ 181.244140] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=768 worker1 # [ 181.259132] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 181.268151] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 181.270391] nix-daemon[855]: reaped child process 1717, status = succeeded client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051... worker1 # [ 181.281286] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) client # farm-top> node-a: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-during-drain worker1 # [ 182.548972] nix-daemon[855]: reaped child process 1726, status = succeeded worker1 # [ 182.664571] nix-daemon[855]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 182.686385] nix-daemon[855]: reaped child process 1745, status = succeeded lb # [ 182.659975] niks3-server[1141]: 2026/09/23 04:45:24 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 182.703844] nix-grpc-daemon[1576]: time=2026-09-23T04:45:24.221Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 182.705830] nix-grpc-daemon[1576]: time=2026-09-23T04:45:24.221Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)" lb # [ 182.701448] niks3-server[1141]: 2026/09/23 04:45:24 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst worker1 # [ 182.745491] nix-grpc-daemon[1576]: time=2026-09-23T04:45:24.264Z level=INFO msg="Uploading 1 narinfos" lb # [ 182.717133] niks3-server[1141]: 2026/09/23 04:45:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign lb # [ 182.726271] niks3-server[1141]: 2026/09/23 04:45:24 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls lb # [ 182.735077] niks3-server[1141]: 2026/09/23 04:45:24 INFO Signed narinfos id=60 count=1 worker1 # [ 182.767117] nix-grpc-daemon[1576]: time=2026-09-23T04:45:24.286Z level=INFO msg="Upload complete. (273ms)" lb # [ 182.738909] niks3-server[1141]: 2026/09/23 04:45:24 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv lb # [ 182.743673] niks3-server[1141]: 2026/09/23 04:45:24 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete lb # [ 182.747833] niks3-server[1141]: 2026/09/23 04:45:24 INFO Completed upload id=60 lb # [ 182.750335] niks3-server[1141]: 2026/09/23 04:45:24 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo worker1 # [ 182.781254] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=1 worker1 # [ 182.794924] nix-daemon[855]: reaped child process 1724, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 5.43 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.18 seconds) worker2: must succeed: pkill -f 'read -t [9]0 x' worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds) client: waiting for success: ! systemctl is-active sw worker2 # [ 183.342086] nix-daemon[843]: reaped child process 1806, status = succeeded worker2 # [ 183.417035] nix-daemon[843]: accepted connection from pid 1983, user nix-grpc-daemon (trusted) worker2 # [ 183.437337] nix-daemon[843]: reaped child process 1986, status = succeeded lb # [ 183.391577] niks3-server[1141]: 2026/09/23 04:45:24 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 183.446314] nix-grpc-daemon[1747]: time=2026-09-23T04:45:24.963Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 183.447785] nix-grpc-daemon[1747]: time=2026-09-23T04:45:24.964Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)" lb # [ 183.431244] niks3-server[1141]: 2026/09/23 04:45:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign worker2 # [ 183.482830] nix-grpc-daemon[1747]: time=2026-09-23T04:45:25.001Z level=INFO msg="Uploading 1 narinfos" lb # [ 183.446144] niks3-server[1141]: 2026/09/23 04:45:24 INFO Signed narinfos id=61 count=1 lb # [ 183.453563] niks3-server[1141]: 2026/09/23 04:45:24 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst lb # [ 183.463903] niks3-server[1141]: 2026/09/23 04:45:24 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls lb # [ 183.467583] niks3-server[1141]: 2026/09/23 04:45:24 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv lb # [ 183.472214] niks3-server[1141]: 2026/09/23 04:45:24 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete lb # [ 183.477741] niks3-server[1141]: 2026/09/23 04:45:25 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo worker2 # [ 183.523081] nix-grpc-daemon[1747]: time=2026-09-23T04:45:25.041Z level=INFO msg="Upload complete. (276ms)" lb # [ 183.484508] niks3-server[1141]: 2026/09/23 04:45:25 INFO Completed upload id=61 worker2 # [ 183.537151] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41730 duration_s=15 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=1 worker2 # [ 183.555533] nix-daemon[843]: reaped child process 1804, status = succeeded client # [ 183.551251] systemd[1]: sw.service: Deactivated successfully. client # [ 183.556558] systemd[1]: sw.service: Consumed 184ms CPU time over 16.757s wall clock time, 41.4M memory peak, 4K written to disk, 7.6K incoming IP traffic, 8.7K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds) client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; } client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.05 seconds) worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 0.06 seconds) lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 184.302128] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:25Z level=info event=signal_exit lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 185.301957] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:26Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:35602 worker1 # [ 185.302181] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:26Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 185.316959] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:26Z level=info event=scheduler_restarting lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 185.342655] nix-grpc-daemon[1665]: ts=2026-09-23T04:45:26Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 185.364965] nix-daemon[843]: reaped child process 1718, status = succeeded lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 185.977596] nix-grpc-daemon[1665]: E0000 00:00:1790138727.495960 1711 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true worker2 # [ 185.994873] nix-grpc-daemon[1665]: E0000 00:00:1790138727.513348 1711 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 187.338177] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 187.342890] systemd[1]: nix-grpc-daemon.service: Consumed 809ms CPU time over 32.317s wall clock time, 36.4M memory peak, 21.8K incoming IP traffic, 31.8K outgoing IP traffic. lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) worker2 # [ 188.341317] systemd[1]: Starting scheduler-token.service... worker2 # [ 188.424144] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 188.429486] systemd[1]: Finished scheduler-token.service. worker2 # [ 188.431491] systemd[1]: scheduler-token.service: Consumed 57ms CPU time over 87ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 188.455675] systemd[1]: Starting Nix worker-protocol over gRPC... lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 188.574191] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:30Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 188.576498] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:30Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 188.580368] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:30Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 188.585697] nix-grpc-daemon[2001]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 188.587123] nix-grpc-daemon[2001]: E0000 00:00:1790138730.103516 2001 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 188.624037] nix-grpc-daemon[2001]: E0000 00:00:1790138730.109654 2001 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 188.626341] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:30Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 4.35 seconds) lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 188.627998] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 188.640056] nix-grpc-daemon[2001]: E0000 00:00:1790138730.120259 2041 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 188.642593] nix-grpc-daemon[2001]: E0000 00:00:1790138730.138133 2041 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 188.647681] nix-grpc-daemon[2001]: E0000 00:00:1790138730.138252 2041 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 188.652471] nix-grpc-daemon[2001]: E0000 00:00:1790138730.156764 2041 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 188.654996] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 188.644807] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:30Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:35602 worker1 # [ 188.645284] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:30Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 21.98 seconds) subtest: queue gauges show a burst while every build blocks lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) client: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]' client # [ 188.955165] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto --impure --expr "map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ \"q0\" \"q1\" \"q2\" \"q3\" ]". client # Running as unit: burst.service; invocation ID: 2de0e452b5364482b62496af09891179 client: (finished: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]', in 0.09 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2 # [ 189.174964] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:30Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:34810 worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker2 # [ 189.190541] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) client # [ 189.190223] nix[1574]: don't know how to build these paths: client # [ 189.191590] nix[1574]: /nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv client # [ 189.193395] nix[1574]: /nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv client # [ 189.196308] nix[1574]: /nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv client # [ 189.198043] nix[1574]: /nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker1 # [ 189.213978] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:30Z level=debug event=want client=5 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv cached=0 worker2 # [ 189.237869] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 worker1 # [ 189.221686] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:30Z level=debug event=assigned drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 189.231726] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:30Z level=debug event=want client=5 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv cached=0 worker1 # [ 189.239803] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:30Z level=debug event=assigned drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 189.254209] nix-grpc-daemon[2001]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 32 ms (attempt 1/5) worker1 # [ 189.248647] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:30Z level=debug event=want client=5 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv cached=0 worker1 # [ 189.254484] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:30Z level=debug event=want client=5 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv cached=0 client # [ 189.256262] nix[1574]: copying 1 paths to 'lb:50051'... worker1 # [ 189.261549] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 client # [ 189.261991] nix[1574]: copying path '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' to 'lb:50051'... worker1 # [ 189.266804] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 189.282983] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 worker2 # [ 189.296281] nix-grpc-daemon[2001]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 197 ms (attempt 2/5) worker1 # [ 189.294858] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 189.432930] nix-daemon[855]: accepted connection from pid 1760, user nix-grpc-daemon (trusted) worker1 # [ 189.452822] nix-daemon[855]: reaped child process 1763, status = succeeded lb # [ 189.425615] niks3-server[1141]: 2026/09/23 04:45:30 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 189.469012] nix-grpc-daemon[1576]: time=2026-09-23T04:45:30.987Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 189.470487] nix-grpc-daemon[1576]: time=2026-09-23T04:45:30.987Z level=INFO msg="Uploading d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv (416B)" worker2 # [ 189.495396] nix-grpc-daemon[2001]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 38 ms (attempt 3/5) lb # [ 189.464778] niks3-server[1141]: 2026/09/23 04:45:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign worker1 # [ 189.504443] nix-grpc-daemon[1576]: time=2026-09-23T04:45:31.023Z level=INFO msg="Uploading 1 narinfos" lb # [ 189.474456] niks3-server[1141]: 2026/09/23 04:45:31 INFO Signed narinfos id=62 count=1 lb # [ 189.484765] niks3-server[1141]: 2026/09/23 04:45:31 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.ls lb # [ 189.491927] niks3-server[1141]: 2026/09/23 04:45:31 INFO Registered completed upload object_key=nar/06ismykzsix0w29371dm1hbzzf3ljf9jn7dyh8skp5wqfkkw0vlk.nar.zst worker2 # [ 189.535405] nix-grpc-daemon[2001]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 693 ms (attempt 4/5) lb # [ 189.503022] niks3-server[1141]: 2026/09/23 04:45:31 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete worker1 # [ 189.539554] nix-grpc-daemon[1576]: time=2026-09-23T04:45:31.058Z level=INFO msg="Upload complete. (239ms)" lb # [ 189.511430] niks3-server[1141]: 2026/09/23 04:45:31 INFO Completed upload id=62 worker1 # [ 189.550707] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=416 lb # [ 189.520874] niks3-server[1141]: 2026/09/23 04:45:31 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.narinfo worker1 # [ 189.568804] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 189.576246] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 189.582347] nix-daemon[855]: reaped child process 1758, status = succeeded client # [ 189.586949] nix[1574]: building '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' on lb:50051... worker1 # [ 189.598370] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 190.233422] nix-grpc-daemon[2001]: disabling binary cache 'https://cache.nixos.org' for 60 seconds lb # [ 190.193238] envoy[891]: conn peer=[2001:db8:1::1]:56484 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9608 tx=4420 ms=5185 worker2 # [ 190.233742] nix-grpc-daemon[2001]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 190.261477] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:34810 client # [ 190.257303] nix[1574]: copying 1 paths to 'lb:50051'... lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds) lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 190.277818] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34810 client # [ 190.263003] nix[1574]: copying path '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' to 'lb:50051'... worker2 # [ 190.293703] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2 # [ 190.322115] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:31Z level=info event=niks3_push_started pid=2062 worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker2 # [ 190.481971] nix-grpc-daemon[2062]: time=2026-09-23T04:45:31.999Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 190.604992] nix-daemon[843]: accepted connection from pid 2079, user nix-grpc-daemon (trusted) worker2 # [ 190.625620] nix-daemon[843]: reaped child process 2082, status = succeeded lb # [ 190.198599] envoy[891]: conn peer=[2001:db8:1::1]:59390 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3038 tx=1578 ms=16392 lb # [ 190.198748] envoy[891]: conn peer=[2001:db8:1::4]:50044 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=3794 tx=500 ms=29974 lb # [ 190.586197] niks3-server[1141]: 2026/09/23 04:45:32 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 190.651404] nix-grpc-daemon[2062]: time=2026-09-23T04:45:32.168Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 190.655368] nix-grpc-daemon[2062]: time=2026-09-23T04:45:32.168Z level=INFO msg="Uploading dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv (416B)" lb # [ 190.634123] niks3-server[1141]: 2026/09/23 04:45:32 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.ls worker2 # [ 190.692946] nix-grpc-daemon[2062]: time=2026-09-23T04:45:32.210Z level=INFO msg="Uploading 1 narinfos" lb # [ 190.650553] niks3-server[1141]: 2026/09/23 04:45:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign lb # [ 190.663202] niks3-server[1141]: 2026/09/23 04:45:32 INFO Signed narinfos id=63 count=1 lb # [ 190.671407] niks3-server[1141]: 2026/09/23 04:45:32 INFO Registered completed upload object_key=nar/06mdwsnf16qmfi86y8r33asi4vcrp5z7kdxqki87psgfpgvvpdki.nar.zst worker2 # [ 190.723398] nix-grpc-daemon[2062]: time=2026-09-23T04:45:32.241Z level=INFO msg="Upload complete. (238ms)" lb # [ 190.679506] niks3-server[1141]: 2026/09/23 04:45:32 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete lb # [ 190.686022] niks3-server[1141]: 2026/09/23 04:45:32 INFO Completed upload id=63 worker2 # [ 190.732319] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34810 duration_s=0 paths=1 nar_bytes_in=416 lb # [ 190.692312] niks3-server[1141]: 2026/09/23 04:45:32 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.narinfo worker2 # [ 190.748919] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 worker2 # [ 190.756471] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) worker2 # [ 190.762111] nix-daemon[843]: reaped child process 2060, status = succeeded client # [ 190.753027] nix[1574]: building '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' on lb:50051... worker2 # [ 190.776070] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) worker1 # [ 191.861602] nix-daemon[855]: reaped child process 1767, status = succeeded worker1 # [ 191.990575] nix-daemon[855]: accepted connection from pid 1795, user nix-grpc-daemon (trusted) worker1 # [ 192.006633] nix-daemon[855]: reaped child process 1798, status = succeeded lb # [ 191.981843] niks3-server[1141]: 2026/09/23 04:45:33 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 192.024425] nix-grpc-daemon[1576]: time=2026-09-23T04:45:33.543Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 192.026204] nix-grpc-daemon[1576]: time=2026-09-23T04:45:33.543Z level=INFO msg="Uploading k6zivhvzgmlyxfmy64fzd759bxlhf9b5-slow-q0 (120B)" lb # [ 192.031039] niks3-server[1141]: 2026/09/23 04:45:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign worker1 # [ 192.074937] nix-grpc-daemon[1576]: time=2026-09-23T04:45:33.594Z level=INFO msg="Uploading 1 narinfos" lb # [ 192.047504] niks3-server[1141]: 2026/09/23 04:45:33 INFO Registered completed upload object_key=log/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv lb # [ 192.063409] niks3-server[1141]: 2026/09/23 04:45:33 INFO Signed narinfos id=64 count=1 worker1 # [ 192.100599] nix-grpc-daemon[1576]: time=2026-09-23T04:45:33.619Z level=INFO msg="Upload complete. (287ms)" lb # [ 192.070230] niks3-server[1141]: 2026/09/23 04:45:33 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.ls lb # [ 192.077777] niks3-server[1141]: 2026/09/23 04:45:33 INFO Registered completed upload object_key=nar/0xr809jj32p5n6mnc6i7lqsz48wrm5f9jp14m77k724041711iq7.nar.zst lb # [ 192.083982] niks3-server[1141]: 2026/09/23 04:45:33 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete worker1 # [ 192.113197] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=2 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv assign_id=7 outputs=1 lb # [ 192.088853] niks3-server[1141]: 2026/09/23 04:45:33 INFO Completed upload id=64 lb # [ 192.091308] niks3-server[1141]: 2026/09/23 04:45:33 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.narinfo worker1 # [ 192.135069] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:33Z level=debug event=assigned drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 192.142037] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 192.148005] nix-daemon[855]: reaped child process 1765, status = succeeded worker1 # [ 192.151460] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 client # [ 192.146390] nix[1574]: copying 1 paths to 'lb:50051'... worker1 # [ 192.156494] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 client # [ 192.152177] nix[1574]: copying path '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' to 'lb:50051'... worker1 # [ 192.172407] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 192.304817] nix-daemon[855]: accepted connection from pid 1802, user nix-grpc-daemon (trusted) worker1 # [ 192.323395] nix-daemon[855]: reaped child process 1805, status = succeeded lb # [ 192.294366] niks3-server[1141]: 2026/09/23 04:45:33 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 192.335596] nix-grpc-daemon[1576]: time=2026-09-23T04:45:33.854Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 192.337300] nix-grpc-daemon[1576]: time=2026-09-23T04:45:33.854Z level=INFO msg="Uploading jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv (416B)" lb # [ 192.323678] niks3-server[1141]: 2026/09/23 04:45:33 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.ls worker1 # [ 192.368363] nix-grpc-daemon[1576]: time=2026-09-23T04:45:33.887Z level=INFO msg="Uploading 1 narinfos" lb # [ 192.335875] niks3-server[1141]: 2026/09/23 04:45:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign lb # [ 192.349140] niks3-server[1141]: 2026/09/23 04:45:33 INFO Signed narinfos id=65 count=1 worker1 # [ 192.392409] nix-grpc-daemon[1576]: time=2026-09-23T04:45:33.911Z level=INFO msg="Upload complete. (214ms)" lb # [ 192.359853] niks3-server[1141]: 2026/09/23 04:45:33 INFO Registered completed upload object_key=nar/1fyfjzlx8vymsfmdv5cv8ymgjkr18fwm737qvys29plwwiwmyvwr.nar.zst worker1 # [ 192.399845] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=416 lb # [ 192.371674] niks3-server[1141]: 2026/09/23 04:45:33 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete lb # [ 192.377460] niks3-server[1141]: 2026/09/23 04:45:33 INFO Completed upload id=65 lb # [ 192.379921] niks3-server[1141]: 2026/09/23 04:45:33 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.narinfo worker1 # [ 192.414611] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 192.420897] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 192.425488] nix-daemon[855]: reaped child process 1800, status = succeeded worker1 # [ 192.436159] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) client # [ 192.427920] nix[1574]: building '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' on lb:50051... worker2 # [ 193.014475] nix-daemon[843]: reaped child process 2086, status = succeeded worker2 # [ 193.111682] nix-daemon[843]: accepted connection from pid 2111, user nix-grpc-daemon (trusted) worker2 # [ 193.129889] nix-daemon[843]: reaped child process 2114, status = succeeded lb # [ 193.091501] niks3-server[1141]: 2026/09/23 04:45:34 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 193.146750] nix-grpc-daemon[2062]: time=2026-09-23T04:45:34.663Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 193.149222] nix-grpc-daemon[2062]: time=2026-09-23T04:45:34.663Z level=INFO msg="Uploading y3qkgrgq2zzpgvi2rf5chq94i627wkdb-slow-q3 (120B)" worker2 # [ 193.187262] nix-grpc-daemon[2062]: time=2026-09-23T04:45:34.705Z level=INFO msg="Uploading 1 narinfos" lb # [ 193.132692] niks3-server[1141]: 2026/09/23 04:45:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign lb # [ 193.146559] niks3-server[1141]: 2026/09/23 04:45:34 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.ls lb # [ 193.156524] niks3-server[1141]: 2026/09/23 04:45:34 INFO Signed narinfos id=66 count=1 worker2 # [ 193.210486] nix-grpc-daemon[2062]: time=2026-09-23T04:45:34.729Z level=INFO msg="Upload complete. (223ms)" worker1 # [ 193.207520] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:34Z level=debug event=assigned drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker=192.168.1.4:50051 clients=1 lb # [ 193.166685] niks3-server[1141]: 2026/09/23 04:45:34 INFO Registered completed upload object_key=nar/1v1grzkvw3yj09x0dkdm2avihvqvnri156qzp768hk85zbw7lkn9.nar.zst worker2 # [ 193.221428] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 duration_s=2 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv assign_id=8 outputs=1 lb # [ 193.179223] niks3-server[1141]: 2026/09/23 04:45:34 INFO Registered completed upload object_key=log/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv lb # [ 193.184713] niks3-server[1141]: 2026/09/23 04:45:34 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete lb # [ 193.189158] niks3-server[1141]: 2026/09/23 04:45:34 INFO Completed upload id=66 worker2 # [ 193.235281] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 lb # [ 193.191582] niks3-server[1141]: 2026/09/23 04:45:34 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.narinfo worker2 # [ 193.239250] nix-daemon[843]: reaped child process 2084, status = succeeded worker2 # [ 193.247167] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:34810 client # [ 193.240080] nix[1574]: copying 1 paths to 'lb:50051'... worker2 # [ 193.258517] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34810 client # [ 193.240324] nix[1574]: copying path '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' to 'lb:50051'... worker2 # [ 193.268440] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) worker2 # [ 193.395498] nix-daemon[843]: accepted connection from pid 2118, user nix-grpc-daemon (trusted) worker2 # [ 193.415907] nix-daemon[843]: reaped child process 2121, status = succeeded lb # [ 193.374362] niks3-server[1141]: 2026/09/23 04:45:34 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 193.427825] nix-grpc-daemon[2062]: time=2026-09-23T04:45:34.946Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 193.429500] nix-grpc-daemon[2062]: time=2026-09-23T04:45:34.946Z level=INFO msg="Uploading pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv (416B)" lb # [ 193.404302] niks3-server[1141]: 2026/09/23 04:45:34 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.ls worker2 # [ 193.459683] nix-grpc-daemon[2062]: time=2026-09-23T04:45:34.977Z level=INFO msg="Uploading 1 narinfos" lb # [ 193.418446] niks3-server[1141]: 2026/09/23 04:45:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign lb # [ 193.429944] niks3-server[1141]: 2026/09/23 04:45:34 INFO Signed narinfos id=67 count=1 lb # [ 193.438959] niks3-server[1141]: 2026/09/23 04:45:34 INFO Registered completed upload object_key=nar/1k21jrz411xc9gkca2pnr9mrjq1s3bg3zhsam3nnv4j066dgzgm7.nar.zst worker2 # [ 193.491734] nix-grpc-daemon[2062]: time=2026-09-23T04:45:35.010Z level=INFO msg="Upload complete. (213ms)" lb # [ 193.448721] niks3-server[1141]: 2026/09/23 04:45:34 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete lb # [ 193.453548] niks3-server[1141]: 2026/09/23 04:45:34 INFO Completed upload id=67 worker2 # [ 193.500247] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34810 duration_s=0 paths=1 nar_bytes_in=416 lb # [ 193.457483] niks3-server[1141]: 2026/09/23 04:45:34 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.narinfo worker2 # [ 193.514103] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 worker2 # [ 193.519866] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) worker2 # [ 193.524892] nix-daemon[843]: reaped child process 2116, status = succeeded client # [ 193.514236] nix[1574]: building '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' on lb:50051... worker2 # [ 193.536678] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) worker1 # [ 194.068791] nix-daemon[855]: reaped child process 1809, status = succeeded worker1 # [ 194.157415] nix-daemon[855]: accepted connection from pid 1834, user nix-grpc-daemon (trusted) worker1 # [ 194.175736] nix-daemon[855]: reaped child process 1837, status = succeeded lb # [ 194.149734] niks3-server[1141]: 2026/09/23 04:45:35 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 194.192380] nix-grpc-daemon[1576]: time=2026-09-23T04:45:35.710Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 194.193883] nix-grpc-daemon[1576]: time=2026-09-23T04:45:35.710Z level=INFO msg="Uploading gjyi8j90pk5flrsqqdhj6h132pcb7n33-slow-q2 (120B)" lb # [ 194.193772] niks3-server[1141]: 2026/09/23 04:45:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign worker1 # [ 194.234269] nix-grpc-daemon[1576]: time=2026-09-23T04:45:35.753Z level=INFO msg="Uploading 1 narinfos" lb # [ 194.208345] niks3-server[1141]: 2026/09/23 04:45:35 INFO Signed narinfos id=68 count=1 lb # [ 194.211142] niks3-server[1141]: 2026/09/23 04:45:35 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.ls lb # [ 194.223027] niks3-server[1141]: 2026/09/23 04:45:35 INFO Registered completed upload object_key=log/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv worker1 # [ 194.265353] nix-grpc-daemon[1576]: time=2026-09-23T04:45:35.784Z level=INFO msg="Upload complete. (216ms)" lb # [ 194.233386] niks3-server[1141]: 2026/09/23 04:45:35 INFO Registered completed upload object_key=nar/15imdx7zyy6mzjjv6qvdzywh0d9w1n6ikc0rq4rq95aw9x6swfzj.nar.zst lb # [ 194.242414] niks3-server[1141]: 2026/09/23 04:45:35 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete worker1 # [ 194.274685] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=1 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv assign_id=9 outputs=1 lb # [ 194.246702] niks3-server[1141]: 2026/09/23 04:45:35 INFO Completed upload id=68 lb # [ 194.249061] niks3-server[1141]: 2026/09/23 04:45:35 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.narinfo worker1 # [ 194.289865] nix-daemon[855]: reaped child process 1807, status = succeeded worker2 # [ 195.375123] nix-daemon[843]: reaped child process 2125, status = succeeded worker2 # [ 195.487714] nix-daemon[843]: accepted connection from pid 2150, user nix-grpc-daemon (trusted) worker2 # [ 195.503937] nix-daemon[843]: reaped child process 2153, status = succeeded lb # [ 195.466892] niks3-server[1141]: 2026/09/23 04:45:36 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 195.520406] nix-grpc-daemon[2062]: time=2026-09-23T04:45:37.038Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 195.522783] nix-grpc-daemon[2062]: time=2026-09-23T04:45:37.040Z level=INFO msg="Uploading l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx-slow-q1 (120B)" lb # [ 195.510793] niks3-server[1141]: 2026/09/23 04:45:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign worker2 # [ 195.566118] nix-grpc-daemon[2062]: time=2026-09-23T04:45:37.084Z level=INFO msg="Uploading 1 narinfos" lb # [ 195.523651] niks3-server[1141]: 2026/09/23 04:45:37 INFO Registered completed upload object_key=nar/0c1c367r3msa4fz6imwp70zxi4fxvrkb86v3hv921wsrf13yd8v3.nar.zst lb # [ 195.534274] niks3-server[1141]: 2026/09/23 04:45:37 INFO Signed narinfos id=69 count=1 lb # [ 195.540604] niks3-server[1141]: 2026/09/23 04:45:37 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.ls worker2 # [ 195.592074] nix-grpc-daemon[2062]: time=2026-09-23T04:45:37.110Z level=INFO msg="Upload complete. (245ms)" lb # [ 195.548527] niks3-server[1141]: 2026/09/23 04:45:37 INFO Registered completed upload object_key=log/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv lb # [ 195.557374] niks3-server[1141]: 2026/09/23 04:45:37 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete lb # [ 195.562956] niks3-server[1141]: 2026/09/23 04:45:37 INFO Completed upload id=69 lb # [ 195.566842] niks3-server[1141]: 2026/09/23 04:45:37 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.narinfo worker2 # [ 195.604713] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 duration_s=2 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv assign_id=10 outputs=1 worker2 # [ 195.620463] nix-daemon[843]: reaped child process 2123, status = succeeded client # [ 195.627111] systemd[1]: burst.service: Deactivated successfully. client # [ 195.632212] systemd[1]: burst.service: Consumed 229ms CPU time over 6.673s wall clock time, 49.7M memory peak, 14K incoming IP traffic, 18.8K outgoing IP traffic. client: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; } client: (finished: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: queue gauges show a burst while every build blocks, in 8.04 seconds) subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1 worker2 # [ 197.009892] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:38Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:34810 worker2 # [ 197.020201] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:34810 worker1 # [ 197.053002] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:38Z level=debug event=want client=6 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=0 worker1 # [ 197.053308] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:38Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 197.053488] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:38Z level=debug event=want client=6 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=0 worker1 # [ 197.053609] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:38Z level=debug event=want client=6 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=0 worker1 # [ 197.097575] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 197.113163] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 197.127484] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 197.142350] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 197.299055] nix-daemon[855]: accepted connection from pid 1854, user nix-grpc-daemon (trusted) worker1 # [ 197.317938] nix-daemon[855]: reaped child process 1857, status = succeeded lb # [ 197.294652] niks3-server[1141]: 2026/09/23 04:45:38 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 197.338615] nix-grpc-daemon[1576]: time=2026-09-23T04:45:38.857Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 197.341005] nix-grpc-daemon[1576]: time=2026-09-23T04:45:38.859Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)" lb # [ 197.337658] niks3-server[1141]: 2026/09/23 04:45:38 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls worker1 # [ 197.379249] nix-grpc-daemon[1576]: time=2026-09-23T04:45:38.898Z level=INFO msg="Uploading 1 narinfos" lb # [ 197.347415] niks3-server[1141]: 2026/09/23 04:45:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign lb # [ 197.355585] niks3-server[1141]: 2026/09/23 04:45:38 INFO Signed narinfos id=70 count=1 lb # [ 197.363560] niks3-server[1141]: 2026/09/23 04:45:38 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst lb # [ 197.377777] niks3-server[1141]: 2026/09/23 04:45:38 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete lb # [ 197.383952] niks3-server[1141]: 2026/09/23 04:45:38 INFO Completed upload id=70 worker1 # [ 197.416219] nix-grpc-daemon[1576]: time=2026-09-23T04:45:38.935Z level=INFO msg="Upload complete. (257ms)" lb # [ 197.395916] niks3-server[1141]: 2026/09/23 04:45:38 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo worker1 # [ 197.428583] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 197.444684] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 197.454818] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 197.459336] nix-daemon[855]: reaped child process 1852, status = succeeded worker1 # [ 197.471886] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 198.556485] nix-daemon[855]: reaped child process 1861, status = succeeded worker1 # [ 198.660620] nix-daemon[855]: accepted connection from pid 1877, user nix-grpc-daemon (trusted) worker1 # [ 198.680832] nix-daemon[855]: reaped child process 1880, status = succeeded lb # [ 198.653467] niks3-server[1141]: 2026/09/23 04:45:40 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 198.696200] nix-grpc-daemon[1576]: time=2026-09-23T04:45:40.214Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 198.697973] nix-grpc-daemon[1576]: time=2026-09-23T04:45:40.214Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)" lb # [ 198.696747] niks3-server[1141]: 2026/09/23 04:45:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign worker1 # [ 198.738082] nix-grpc-daemon[1576]: time=2026-09-23T04:45:40.257Z level=INFO msg="Uploading 1 narinfos" lb # [ 198.710824] niks3-server[1141]: 2026/09/23 04:45:40 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst lb # [ 198.719972] niks3-server[1141]: 2026/09/23 04:45:40 INFO Signed narinfos id=71 count=1 lb # [ 198.724844] niks3-server[1141]: 2026/09/23 04:45:40 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv lb # [ 198.732475] niks3-server[1141]: 2026/09/23 04:45:40 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls worker1 # [ 198.765141] nix-grpc-daemon[1576]: time=2026-09-23T04:45:40.284Z level=INFO msg="Upload complete. (275ms)" lb # [ 198.735524] niks3-server[1141]: 2026/09/23 04:45:40 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete lb # [ 198.740226] niks3-server[1141]: 2026/09/23 04:45:40 INFO Completed upload id=71 lb # [ 198.742573] niks3-server[1141]: 2026/09/23 04:45:40 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo worker1 # [ 198.777195] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=11 outputs=1 worker1 # [ 198.800349] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:40Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 198.810965] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 198.814883] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:40Z level=debug event=want client=6 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=0 worker1 # [ 198.818457] nix-daemon[855]: reaped child process 1859, status = succeeded worker1 # [ 198.832429] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 198.845738] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 198.858412] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 199.002894] nix-daemon[855]: accepted connection from pid 1886, user nix-grpc-daemon (trusted) worker1 # [ 199.021643] nix-daemon[855]: reaped child process 1889, status = succeeded lb # [ 198.990099] niks3-server[1141]: 2026/09/23 04:45:40 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 199.030799] nix-grpc-daemon[1576]: time=2026-09-23T04:45:40.550Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 199.032640] nix-grpc-daemon[1576]: time=2026-09-23T04:45:40.550Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)" lb # [ 199.019149] niks3-server[1141]: 2026/09/23 04:45:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign worker1 # [ 199.057313] nix-grpc-daemon[1576]: time=2026-09-23T04:45:40.576Z level=INFO msg="Uploading 1 narinfos" lb # [ 199.032818] niks3-server[1141]: 2026/09/23 04:45:40 INFO Signed narinfos id=72 count=1 lb # [ 199.040441] niks3-server[1141]: 2026/09/23 04:45:40 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls worker1 # [ 199.083682] nix-grpc-daemon[1576]: time=2026-09-23T04:45:40.603Z level=INFO msg="Upload complete. (212ms)" lb # [ 199.052749] niks3-server[1141]: 2026/09/23 04:45:40 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst lb # [ 199.057626] niks3-server[1141]: 2026/09/23 04:45:40 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete worker1 # [ 199.092262] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 199.063265] niks3-server[1141]: 2026/09/23 04:45:40 INFO Completed upload id=72 lb # [ 199.067354] niks3-server[1141]: 2026/09/23 04:45:40 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo worker1 # [ 199.105277] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 199.110935] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 199.115714] nix-daemon[855]: reaped child process 1884, status = succeeded worker1 # [ 199.125855] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) lb # [ 200.191420] envoy[891]: conn peer=[2001:db8:1::1]:46316 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9662 tx=5231 ms=6460 worker1 # [ 200.620419] nix-daemon[855]: reaped child process 1893, status = succeeded worker1 # [ 200.710830] nix-daemon[855]: accepted connection from pid 1909, user nix-grpc-daemon (trusted) worker1 # [ 200.731824] nix-daemon[855]: reaped child process 1912, status = succeeded lb # [ 200.703152] niks3-server[1141]: 2026/09/23 04:45:42 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 200.748311] nix-grpc-daemon[1576]: time=2026-09-23T04:45:42.264Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 200.750023] nix-grpc-daemon[1576]: time=2026-09-23T04:45:42.264Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)" lb # [ 200.742048] niks3-server[1141]: 2026/09/23 04:45:42 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst lb # [ 200.755366] niks3-server[1141]: 2026/09/23 04:45:42 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker1 # [ 200.798632] nix-grpc-daemon[1576]: time=2026-09-23T04:45:42.317Z level=INFO msg="Uploading 1 narinfos" lb # [ 200.768299] niks3-server[1141]: 2026/09/23 04:45:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign lb # [ 200.778041] niks3-server[1141]: 2026/09/23 04:45:42 INFO Signed narinfos id=73 count=1 lb # [ 200.783404] niks3-server[1141]: 2026/09/23 04:45:42 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls worker1 # [ 200.827397] nix-grpc-daemon[1576]: time=2026-09-23T04:45:42.346Z level=INFO msg="Upload complete. (289ms)" lb # [ 200.796351] niks3-server[1141]: 2026/09/23 04:45:42 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete lb # [ 200.799701] niks3-server[1141]: 2026/09/23 04:45:42 INFO Completed upload id=73 lb # [ 200.804723] niks3-server[1141]: 2026/09/23 04:45:42 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo worker1 # [ 200.845486] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=12 outputs=1 worker1 # [ 200.860326] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:42Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 200.867492] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 200.871198] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:42Z level=debug event=want client=6 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=0 worker1 # [ 200.875286] nix-daemon[855]: reaped child process 1891, status = succeeded worker1 # [ 200.878682] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 200.891149] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 200.902277] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 201.040943] nix-daemon[855]: accepted connection from pid 1918, user nix-grpc-daemon (trusted) worker1 # [ 201.060528] nix-daemon[855]: reaped child process 1921, status = succeeded lb # [ 201.031571] niks3-server[1141]: 2026/09/23 04:45:42 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 201.074533] nix-grpc-daemon[1576]: time=2026-09-23T04:45:42.593Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 201.076484] nix-grpc-daemon[1576]: time=2026-09-23T04:45:42.593Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)" lb # [ 201.062698] niks3-server[1141]: 2026/09/23 04:45:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign worker1 # [ 201.101319] nix-grpc-daemon[1576]: time=2026-09-23T04:45:42.620Z level=INFO msg="Uploading 1 narinfos" lb # [ 201.077015] niks3-server[1141]: 2026/09/23 04:45:42 INFO Signed narinfos id=74 count=1 lb # [ 201.084869] niks3-server[1141]: 2026/09/23 04:45:42 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst worker1 # [ 201.130476] nix-grpc-daemon[1576]: time=2026-09-23T04:45:42.649Z level=INFO msg="Upload complete. (215ms)" lb # [ 201.098291] niks3-server[1141]: 2026/09/23 04:45:42 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls lb # [ 201.103663] niks3-server[1141]: 2026/09/23 04:45:42 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete lb # [ 201.108779] niks3-server[1141]: 2026/09/23 04:45:42 INFO Completed upload id=74 worker1 # [ 201.140271] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 201.114040] niks3-server[1141]: 2026/09/23 04:45:42 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo worker1 # [ 201.152274] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 201.159061] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 201.164263] nix-daemon[855]: reaped child process 1916, status = succeeded worker1 # [ 201.177146] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 202.228657] nix-daemon[855]: reaped child process 1925, status = succeeded worker1 # [ 202.381611] nix-daemon[855]: accepted connection from pid 1941, user nix-grpc-daemon (trusted) worker1 # [ 202.405810] nix-daemon[855]: reaped child process 1944, status = succeeded lb # [ 202.375394] niks3-server[1141]: 2026/09/23 04:45:43 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 202.416525] nix-grpc-daemon[1576]: time=2026-09-23T04:45:43.935Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 202.418023] nix-grpc-daemon[1576]: time=2026-09-23T04:45:43.935Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)" lb # [ 202.435507] niks3-server[1141]: 2026/09/23 04:45:43 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst lb # [ 202.457186] niks3-server[1141]: 2026/09/23 04:45:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign worker1 # [ 202.495933] nix-grpc-daemon[1576]: time=2026-09-23T04:45:44.014Z level=INFO msg="Uploading 1 narinfos" lb # [ 202.469324] niks3-server[1141]: 2026/09/23 04:45:43 INFO Signed narinfos id=75 count=1 lb # [ 202.478144] niks3-server[1141]: 2026/09/23 04:45:44 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls lb # [ 202.495399] niks3-server[1141]: 2026/09/23 04:45:44 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete worker1 # [ 202.534706] nix-grpc-daemon[1576]: time=2026-09-23T04:45:44.054Z level=INFO msg="Upload complete. (344ms)" lb # [ 202.505864] niks3-server[1141]: 2026/09/23 04:45:44 INFO Completed upload id=75 lb # [ 202.518831] niks3-server[1141]: 2026/09/23 04:45:44 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo lb # [ 202.529535] niks3-server[1141]: 2026/09/23 04:45:44 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker1 # [ 202.562507] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=13 outputs=1 worker1 # [ 202.580751] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:44Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 202.594676] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 202.604298] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:44Z level=debug event=want client=6 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=0 worker1 # [ 202.607675] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 202.614985] nix-daemon[855]: reaped child process 1923, status = succeeded worker1 # [ 202.618963] nix-grpc-daemon[1576]: time=2026-09-23T04:45:44.138Z level=INFO msg="All 1 paths already cached" worker1 # [ 202.632488] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 202.646480] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 202.848408] nix-daemon[855]: accepted connection from pid 1950, user nix-grpc-daemon (trusted) worker1 # [ 202.871532] nix-daemon[855]: reaped child process 1953, status = succeeded lb # [ 202.843866] niks3-server[1141]: 2026/09/23 04:45:44 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 202.888796] nix-grpc-daemon[1576]: time=2026-09-23T04:45:44.407Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 202.890573] nix-grpc-daemon[1576]: time=2026-09-23T04:45:44.407Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)" lb # [ 202.890557] niks3-server[1141]: 2026/09/23 04:45:44 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst worker1 # [ 202.937393] nix-grpc-daemon[1576]: time=2026-09-23T04:45:44.456Z level=INFO msg="Uploading 1 narinfos" lb # [ 202.910371] niks3-server[1141]: 2026/09/23 04:45:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign lb # [ 202.921313] niks3-server[1141]: 2026/09/23 04:45:44 INFO Signed narinfos id=76 count=1 lb # [ 202.927710] niks3-server[1141]: 2026/09/23 04:45:44 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls lb # [ 202.951002] niks3-server[1141]: 2026/09/23 04:45:44 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete worker1 # [ 202.992591] nix-grpc-daemon[1576]: time=2026-09-23T04:45:44.511Z level=INFO msg="Upload complete. (328ms)" lb # [ 202.962406] niks3-server[1141]: 2026/09/23 04:45:44 INFO Completed upload id=76 lb # [ 202.965499] niks3-server[1141]: 2026/09/23 04:45:44 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo worker1 # [ 203.004555] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 203.020668] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 203.038479] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 203.048341] nix-daemon[855]: reaped child process 1948, status = succeeded worker1 # [ 203.068414] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 204.230419] nix-daemon[855]: reaped child process 1957, status = succeeded worker1 # [ 204.333073] nix-daemon[855]: accepted connection from pid 1973, user nix-grpc-daemon (trusted) worker1 # [ 204.358271] nix-daemon[855]: reaped child process 1976, status = succeeded lb # [ 204.335929] niks3-server[1141]: 2026/09/23 04:45:45 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 204.378809] nix-grpc-daemon[1576]: time=2026-09-23T04:45:45.895Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 204.382174] nix-grpc-daemon[1576]: time=2026-09-23T04:45:45.895Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)" lb # [ 204.393578] niks3-server[1141]: 2026/09/23 04:45:45 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls lb # [ 204.405553] niks3-server[1141]: 2026/09/23 04:45:45 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv lb # [ 204.413765] niks3-server[1141]: 2026/09/23 04:45:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign worker1 # [ 204.447269] nix-grpc-daemon[1576]: time=2026-09-23T04:45:45.965Z level=INFO msg="Uploading 1 narinfos" lb # [ 204.424930] niks3-server[1141]: 2026/09/23 04:45:45 INFO Signed narinfos id=77 count=1 lb # [ 204.431873] niks3-server[1141]: 2026/09/23 04:45:45 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst lb # [ 204.448112] niks3-server[1141]: 2026/09/23 04:45:45 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete worker1 # [ 204.485737] nix-grpc-daemon[1576]: time=2026-09-23T04:45:46.004Z level=INFO msg="Upload complete. (395ms)" lb # [ 204.456186] niks3-server[1141]: 2026/09/23 04:45:45 INFO Completed upload id=77 lb # [ 204.459616] niks3-server[1141]: 2026/09/23 04:45:45 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo worker1 # [ 204.505400] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=14 outputs=1 worker1 # [ 204.528474] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:46Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 204.533817] nix-daemon[855]: reaped child process 1955, status = succeeded worker1 # [ 204.547978] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:46Z level=debug event=want client=6 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=0 worker1 # [ 204.554524] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 204.573167] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 204.590887] nix-grpc-daemon[1576]: time=2026-09-23T04:45:46.110Z level=INFO msg="All 1 paths already cached" worker1 # [ 204.604489] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 204.621494] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 204.835502] nix-daemon[855]: accepted connection from pid 1982, user nix-grpc-daemon (trusted) worker1 # [ 204.863947] nix-daemon[855]: reaped child process 1985, status = succeeded lb # [ 204.828849] niks3-server[1141]: 2026/09/23 04:45:46 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 204.872203] nix-grpc-daemon[1576]: time=2026-09-23T04:45:46.391Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 204.873799] nix-grpc-daemon[1576]: time=2026-09-23T04:45:46.391Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)" lb # [ 204.881662] niks3-server[1141]: 2026/09/23 04:45:46 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst worker1 # [ 204.923672] nix-grpc-daemon[1576]: time=2026-09-23T04:45:46.442Z level=INFO msg="Uploading 1 narinfos" lb # [ 204.896764] niks3-server[1141]: 2026/09/23 04:45:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign lb # [ 204.904495] niks3-server[1141]: 2026/09/23 04:45:46 INFO Signed narinfos id=78 count=1 lb # [ 204.914328] niks3-server[1141]: 2026/09/23 04:45:46 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls lb # [ 204.924454] niks3-server[1141]: 2026/09/23 04:45:46 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete worker1 # [ 204.963277] nix-grpc-daemon[1576]: time=2026-09-23T04:45:46.482Z level=INFO msg="Upload complete. (318ms)" lb # [ 204.935742] niks3-server[1141]: 2026/09/23 04:45:46 INFO Completed upload id=78 lb # [ 204.942281] niks3-server[1141]: 2026/09/23 04:45:46 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo worker1 # [ 204.976582] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 204.993013] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 205.013263] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 205.023335] nix-daemon[855]: reaped child process 1980, status = succeeded worker1 # [ 205.044544] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 206.360712] nix-daemon[855]: reaped child process 1989, status = succeeded worker1 # [ 206.434767] nix-daemon[855]: accepted connection from pid 2005, user nix-grpc-daemon (trusted) worker1 # [ 206.453197] nix-daemon[855]: reaped child process 2008, status = succeeded lb # [ 206.429817] niks3-server[1141]: 2026/09/23 04:45:47 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 206.473760] nix-grpc-daemon[1576]: time=2026-09-23T04:45:47.991Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 206.476320] nix-grpc-daemon[1576]: time=2026-09-23T04:45:47.991Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)" lb # [ 206.473990] niks3-server[1141]: 2026/09/23 04:45:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign worker1 # [ 206.516972] nix-grpc-daemon[1576]: time=2026-09-23T04:45:48.036Z level=INFO msg="Uploading 1 narinfos" lb # [ 206.488848] niks3-server[1141]: 2026/09/23 04:45:48 INFO Signed narinfos id=79 count=1 lb # [ 206.491297] niks3-server[1141]: 2026/09/23 04:45:48 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst lb # [ 206.506390] niks3-server[1141]: 2026/09/23 04:45:48 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker1 # [ 206.544335] nix-grpc-daemon[1576]: time=2026-09-23T04:45:48.063Z level=INFO msg="Upload complete. (280ms)" lb # [ 206.514231] niks3-server[1141]: 2026/09/23 04:45:48 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete lb # [ 206.519882] niks3-server[1141]: 2026/09/23 04:45:48 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls lb # [ 206.527333] niks3-server[1141]: 2026/09/23 04:45:48 INFO Completed upload id=79 lb # [ 206.530319] niks3-server[1141]: 2026/09/23 04:45:48 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo worker1 # [ 206.562015] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=15 outputs=1 worker1 # [ 206.585029] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:48Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 206.589930] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 206.595376] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:48Z level=debug event=want client=6 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=0 worker1 # [ 206.600347] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 206.605188] nix-daemon[855]: reaped child process 1987, status = succeeded worker1 # [ 206.606958] nix-grpc-daemon[1576]: time=2026-09-23T04:45:48.123Z level=INFO msg="All 1 paths already cached" worker1 # [ 206.610452] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 206.624234] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 206.750666] nix-daemon[855]: accepted connection from pid 2014, user nix-grpc-daemon (trusted) worker1 # [ 206.768654] nix-daemon[855]: reaped child process 2017, status = succeeded lb # [ 206.743748] niks3-server[1141]: 2026/09/23 04:45:48 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 206.785007] nix-grpc-daemon[1576]: time=2026-09-23T04:45:48.304Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 206.786894] nix-grpc-daemon[1576]: time=2026-09-23T04:45:48.304Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)" lb # [ 206.772927] niks3-server[1141]: 2026/09/23 04:45:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign worker1 # [ 206.810840] nix-grpc-daemon[1576]: time=2026-09-23T04:45:48.330Z level=INFO msg="Uploading 1 narinfos" lb # [ 206.788469] niks3-server[1141]: 2026/09/23 04:45:48 INFO Signed narinfos id=80 count=1 lb # [ 206.795116] niks3-server[1141]: 2026/09/23 04:45:48 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst lb # [ 206.809992] niks3-server[1141]: 2026/09/23 04:45:48 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete worker1 # [ 206.848879] nix-grpc-daemon[1576]: time=2026-09-23T04:45:48.368Z level=INFO msg="Upload complete. (218ms)" lb # [ 206.818323] niks3-server[1141]: 2026/09/23 04:45:48 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls worker1 # [ 206.856155] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 206.826008] niks3-server[1141]: 2026/09/23 04:45:48 INFO Completed upload id=80 lb # [ 206.829165] niks3-server[1141]: 2026/09/23 04:45:48 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo worker1 # [ 206.870947] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 206.877368] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 206.882488] nix-daemon[855]: reaped child process 2012, status = succeeded worker1 # [ 206.893404] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 207.792254] nix-daemon[855]: reaped child process 2021, status = succeeded worker1 # [ 207.866628] nix-daemon[855]: accepted connection from pid 2037, user nix-grpc-daemon (trusted) worker1 # [ 207.885311] nix-daemon[855]: reaped child process 2040, status = succeeded lb # [ 207.861469] niks3-server[1141]: 2026/09/23 04:45:49 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 207.904408] nix-grpc-daemon[1576]: time=2026-09-23T04:45:49.423Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 207.906022] nix-grpc-daemon[1576]: time=2026-09-23T04:45:49.423Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)" lb # [ 207.908777] niks3-server[1141]: 2026/09/23 04:45:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign worker1 # [ 207.948545] nix-grpc-daemon[1576]: time=2026-09-23T04:45:49.467Z level=INFO msg="Uploading 1 narinfos" lb # [ 207.919723] niks3-server[1141]: 2026/09/23 04:45:49 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv lb # [ 207.928741] niks3-server[1141]: 2026/09/23 04:45:49 INFO Signed narinfos id=81 count=1 lb # [ 207.934790] niks3-server[1141]: 2026/09/23 04:45:49 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst lb # [ 207.943732] niks3-server[1141]: 2026/09/23 04:45:49 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete lb # [ 207.948377] niks3-server[1141]: 2026/09/23 04:45:49 INFO Completed upload id=81 worker1 # [ 207.973744] nix-grpc-daemon[1576]: time=2026-09-23T04:45:49.493Z level=INFO msg="Upload complete. (268ms)" lb # [ 207.950526] niks3-server[1141]: 2026/09/23 04:45:49 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo lb # [ 207.955354] niks3-server[1141]: 2026/09/23 04:45:49 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls worker1 # [ 207.985278] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=16 outputs=1 worker1 # [ 208.004866] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:49Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 208.009850] nix-daemon[855]: reaped child process 2019, status = succeeded worker1 # [ 208.028806] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:49Z level=debug event=want client=6 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=0 worker1 # [ 208.036273] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 208.046584] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 208.060084] nix-grpc-daemon[1576]: time=2026-09-23T04:45:49.579Z level=INFO msg="All 2 paths already cached" worker1 # [ 208.069352] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 208.083003] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 208.219106] nix-daemon[855]: accepted connection from pid 2046, user nix-grpc-daemon (trusted) worker1 # [ 208.238049] nix-daemon[855]: reaped child process 2049, status = succeeded lb # [ 208.211885] niks3-server[1141]: 2026/09/23 04:45:49 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 208.252531] nix-grpc-daemon[1576]: time=2026-09-23T04:45:49.772Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 208.254505] nix-grpc-daemon[1576]: time=2026-09-23T04:45:49.773Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)" lb # [ 208.243565] niks3-server[1141]: 2026/09/23 04:45:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign worker1 # [ 208.281888] nix-grpc-daemon[1576]: time=2026-09-23T04:45:49.801Z level=INFO msg="Uploading 1 narinfos" lb # [ 208.257009] niks3-server[1141]: 2026/09/23 04:45:49 INFO Signed narinfos id=82 count=1 lb # [ 208.259698] niks3-server[1141]: 2026/09/23 04:45:49 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls lb # [ 208.274603] niks3-server[1141]: 2026/09/23 04:45:49 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete lb # [ 208.278133] niks3-server[1141]: 2026/09/23 04:45:49 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst lb # [ 208.283168] niks3-server[1141]: 2026/09/23 04:45:49 INFO Completed upload id=82 worker1 # [ 208.315194] nix-grpc-daemon[1576]: time=2026-09-23T04:45:49.834Z level=INFO msg="Upload complete. (219ms)" lb # [ 208.287803] niks3-server[1141]: 2026/09/23 04:45:49 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo worker1 # [ 208.323559] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 208.339626] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 208.347616] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 208.358547] nix-daemon[855]: reaped child process 2044, status = succeeded worker1 # [ 208.369510] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 209.622079] nix-daemon[855]: reaped child process 2053, status = succeeded worker1 # [ 209.730056] nix-daemon[855]: accepted connection from pid 2069, user nix-grpc-daemon (trusted) worker1 # [ 209.748097] nix-daemon[855]: reaped child process 2072, status = succeeded lb # [ 209.723046] niks3-server[1141]: 2026/09/23 04:45:51 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 209.766569] nix-grpc-daemon[1576]: time=2026-09-23T04:45:51.284Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 209.768259] nix-grpc-daemon[1576]: time=2026-09-23T04:45:51.284Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)" lb # [ 209.764767] niks3-server[1141]: 2026/09/23 04:45:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign worker1 # [ 209.806737] nix-grpc-daemon[1576]: time=2026-09-23T04:45:51.326Z level=INFO msg="Uploading 1 narinfos" lb # [ 209.777781] niks3-server[1141]: 2026/09/23 04:45:51 INFO Signed narinfos id=83 count=1 lb # [ 209.785731] niks3-server[1141]: 2026/09/23 04:45:51 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv lb # [ 209.793892] niks3-server[1141]: 2026/09/23 04:45:51 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst worker1 # [ 209.834099] nix-grpc-daemon[1576]: time=2026-09-23T04:45:51.353Z level=INFO msg="Upload complete. (251ms)" lb # [ 209.805252] niks3-server[1141]: 2026/09/23 04:45:51 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls lb # [ 209.810539] niks3-server[1141]: 2026/09/23 04:45:51 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete lb # [ 209.815475] niks3-server[1141]: 2026/09/23 04:45:51 INFO Completed upload id=83 lb # [ 209.817976] niks3-server[1141]: 2026/09/23 04:45:51 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo worker1 # [ 209.848725] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=17 outputs=1 worker1 # [ 209.863335] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:51Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 209.867851] nix-daemon[855]: reaped child process 2051, status = succeeded worker1 # [ 209.889172] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 209.903463] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 209.917715] nix-grpc-daemon[1576]: time=2026-09-23T04:45:51.437Z level=INFO msg="All 2 paths already cached" worker1 # [ 209.927471] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 209.941459] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 210.094747] nix-daemon[855]: accepted connection from pid 2076, user nix-grpc-daemon (trusted) worker1 # [ 210.113491] nix-daemon[855]: reaped child process 2079, status = succeeded lb # [ 210.085667] niks3-server[1141]: 2026/09/23 04:45:51 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 210.126629] nix-grpc-daemon[1576]: time=2026-09-23T04:45:51.645Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 210.128321] nix-grpc-daemon[1576]: time=2026-09-23T04:45:51.645Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)" lb # [ 210.114936] niks3-server[1141]: 2026/09/23 04:45:51 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst worker1 # [ 210.159007] nix-grpc-daemon[1576]: time=2026-09-23T04:45:51.678Z level=INFO msg="Uploading 1 narinfos" lb # [ 210.127384] niks3-server[1141]: 2026/09/23 04:45:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign lb # [ 210.139572] niks3-server[1141]: 2026/09/23 04:45:51 INFO Signed narinfos id=84 count=1 lb # [ 210.147446] niks3-server[1141]: 2026/09/23 04:45:51 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls lb # [ 210.155681] niks3-server[1141]: 2026/09/23 04:45:51 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete lb # [ 210.160487] niks3-server[1141]: 2026/09/23 04:45:51 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo worker1 # [ 210.192769] nix-grpc-daemon[1576]: time=2026-09-23T04:45:51.712Z level=INFO msg="Upload complete. (234ms)" lb # [ 210.163314] niks3-server[1141]: 2026/09/23 04:45:51 INFO Completed upload id=84 worker1 # [ 210.200212] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 210.215977] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 210.224315] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 210.229144] nix-daemon[855]: reaped child process 2074, status = succeeded worker1 # [ 210.243417] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 211.530478] nix-daemon[855]: reaped child process 2083, status = succeeded worker1 # [ 211.598586] nix-daemon[855]: accepted connection from pid 2099, user nix-grpc-daemon (trusted) worker1 # [ 211.616948] nix-daemon[855]: reaped child process 2102, status = succeeded lb # [ 211.589785] niks3-server[1141]: 2026/09/23 04:45:53 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 211.636414] nix-grpc-daemon[1576]: time=2026-09-23T04:45:53.153Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 211.638137] nix-grpc-daemon[1576]: time=2026-09-23T04:45:53.153Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)" lb # [ 211.643488] niks3-server[1141]: 2026/09/23 04:45:53 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv lb # [ 211.654058] niks3-server[1141]: 2026/09/23 04:45:53 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls worker1 # [ 211.695887] nix-grpc-daemon[1576]: time=2026-09-23T04:45:53.214Z level=INFO msg="Uploading 1 narinfos" lb # [ 211.663461] niks3-server[1141]: 2026/09/23 04:45:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign lb # [ 211.671890] niks3-server[1141]: 2026/09/23 04:45:53 INFO Signed narinfos id=85 count=1 lb # [ 211.679443] niks3-server[1141]: 2026/09/23 04:45:53 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst lb # [ 211.689052] niks3-server[1141]: 2026/09/23 04:45:53 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete lb # [ 211.694256] niks3-server[1141]: 2026/09/23 04:45:53 INFO Completed upload id=85 worker1 # [ 211.718063] nix-grpc-daemon[1576]: time=2026-09-23T04:45:53.237Z level=INFO msg="Upload complete. (260ms)" lb # [ 211.697714] niks3-server[1141]: 2026/09/23 04:45:53 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo worker1 # [ 211.731350] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=18 outputs=1 worker1 # [ 211.748359] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:53Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 211.754220] nix-daemon[855]: reaped child process 2081, status = succeeded worker1 # [ 211.772737] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 211.787622] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 211.802003] nix-grpc-daemon[1576]: time=2026-09-23T04:45:53.321Z level=INFO msg="All 2 paths already cached" worker1 # [ 211.812111] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 211.825543] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 211.973830] nix-daemon[855]: accepted connection from pid 2106, user nix-grpc-daemon (trusted) worker1 # [ 211.991034] nix-daemon[855]: reaped child process 2109, status = succeeded lb # [ 211.966336] niks3-server[1141]: 2026/09/23 04:45:53 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 212.007714] nix-grpc-daemon[1576]: time=2026-09-23T04:45:53.526Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 212.009542] nix-grpc-daemon[1576]: time=2026-09-23T04:45:53.526Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)" lb # [ 211.994918] niks3-server[1141]: 2026/09/23 04:45:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/86/sign worker1 # [ 212.033278] nix-grpc-daemon[1576]: time=2026-09-23T04:45:53.552Z level=INFO msg="Uploading 1 narinfos" lb # [ 212.007582] niks3-server[1141]: 2026/09/23 04:45:53 INFO Signed narinfos id=86 count=1 lb # [ 212.017143] niks3-server[1141]: 2026/09/23 04:45:53 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls lb # [ 212.028961] niks3-server[1141]: 2026/09/23 04:45:53 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst worker1 # [ 212.068444] nix-grpc-daemon[1576]: time=2026-09-23T04:45:53.587Z level=INFO msg="Upload complete. (228ms)" lb # [ 212.036673] niks3-server[1141]: 2026/09/23 04:45:53 INFO Received complete upload request method=POST path=/api/pending_closures/86/complete lb # [ 212.045408] niks3-server[1141]: 2026/09/23 04:45:53 INFO Completed upload id=86 worker1 # [ 212.078160] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 212.047899] niks3-server[1141]: 2026/09/23 04:45:53 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo worker1 # [ 212.092448] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 212.098635] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 212.103351] nix-daemon[855]: reaped child process 2104, status = succeeded worker1 # [ 212.113349] nix-daemon[855]: accepted connection from pid 1475, user nix-grpc-daemon (trusted) worker1 # [ 212.974980] nix-daemon[855]: reaped child process 2113, status = succeeded worker1 # [ 213.108708] nix-daemon[855]: accepted connection from pid 2129, user nix-grpc-daemon (trusted) worker1 # [ 213.124431] nix-daemon[855]: reaped child process 2132, status = succeeded lb # [ 213.100834] niks3-server[1141]: 2026/09/23 04:45:54 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 213.144194] nix-grpc-daemon[1576]: time=2026-09-23T04:45:54.662Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 213.146089] nix-grpc-daemon[1576]: time=2026-09-23T04:45:54.662Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)" lb # [ 213.141790] niks3-server[1141]: 2026/09/23 04:45:54 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker1 # [ 213.188081] nix-grpc-daemon[1576]: time=2026-09-23T04:45:54.707Z level=INFO msg="Uploading 1 narinfos" lb # [ 213.157289] niks3-server[1141]: 2026/09/23 04:45:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/87/sign lb # [ 213.165757] niks3-server[1141]: 2026/09/23 04:45:54 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst lb # [ 213.174294] niks3-server[1141]: 2026/09/23 04:45:54 INFO Signed narinfos id=87 count=1 lb # [ 213.177479] niks3-server[1141]: 2026/09/23 04:45:54 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls worker1 # [ 213.216126] nix-grpc-daemon[1576]: time=2026-09-23T04:45:54.735Z level=INFO msg="Upload complete. (292ms)" lb # [ 213.184882] niks3-server[1141]: 2026/09/23 04:45:54 INFO Received complete upload request method=POST path=/api/pending_closures/87/complete lb # [ 213.188608] niks3-server[1141]: 2026/09/23 04:45:54 INFO Completed upload id=87 lb # [ 213.191936] niks3-server[1141]: 2026/09/23 04:45:54 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo worker1 # [ 213.229946] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32848 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=19 outputs=1 worker1 # [ 213.249240] nix-daemon[855]: reaped child process 2111, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 16.53 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]' client # [ 213.425566] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]". client # Running as unit: gpu.service; invocation ID: b1ffe0dbd9234dfdbcc040b07b5a5ad2 client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics client # [ 213.632964] nix[1647]: don't know how to build these paths: client # [ 213.634257] nix[1647]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv worker1 # [ 213.640316] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:55Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:32848 worker2 # [ 213.659347] nix-grpc-daemon[2001]: ts=2026-09-23T04:45:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:34810 worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker1 # [ 213.676142] nix-grpc-daemon[1475]: ts=2026-09-23T04:45:55Z level=debug event=want client=7 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=0 client # [ 213.676140] nix[1647]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip}, waiting up to 30s for a worker lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) client: waiting for success: systemctl show -p Result gpu | grep -q exit-code lb # [ 220.191180] envoy[891]: conn peer=[2001:db8:1::1]:51932 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27179 tx=12334 ms=16249 client # [ 243.683564] nix[1647]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip} client # [ 243.698930] nix[1647]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed client # [ 243.730559] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE client # [ 243.738742] systemd[1]: gpu.service: Failed with result 'exit-code'. client # [ 243.742545] systemd[1]: gpu.service: Consumed 203ms CPU time over 30.310s wall clock time, 35.6M memory peak, 5.6K incoming IP traffic, 5.2K outgoing IP traffic. client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.08 seconds) client: must succeed: journalctl -u gpu -o cat client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 47.34 seconds) subtest: low disk drains a worker and builds go to the other worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill worker1 # [ 245.101363] nix-grpc-daemon[1475]: ts=2026-09-23T04:46:26Z level=info event=unhealthy reason=min_free worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.14 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free' worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 0.03 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2 client # don't know how to build these paths: client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker1 # [ 245.515942] nix-grpc-daemon[1475]: ts=2026-09-23T04:46:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 245.525519] nix-grpc-daemon[1475]: ts=2026-09-23T04:46:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:32848 worker1 # [ 245.545180] nix-grpc-daemon[1475]: ts=2026-09-23T04:46:27Z level=debug event=want client=8 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=0 worker2 # [ 245.562494] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 worker1 # [ 245.551466] nix-grpc-daemon[1475]: ts=2026-09-23T04:46:27Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 245.578247] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:34810 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'... worker2 # [ 245.594309] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34810 worker2 # [ 245.611438] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) worker2 # [ 245.784903] nix-daemon[843]: accepted connection from pid 2161, user nix-grpc-daemon (trusted) worker2 # [ 245.803674] nix-daemon[843]: reaped child process 2164, status = succeeded lb # [ 245.767593] niks3-server[1141]: 2026/09/23 04:46:27 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 245.824599] nix-grpc-daemon[2062]: time=2026-09-23T04:46:27.342Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 245.826905] nix-grpc-daemon[2062]: time=2026-09-23T04:46:27.342Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)" lb # [ 245.803665] niks3-server[1141]: 2026/09/23 04:46:27 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls lb # [ 245.818992] niks3-server[1141]: 2026/09/23 04:46:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/88/sign worker2 # [ 245.870657] nix-grpc-daemon[2062]: time=2026-09-23T04:46:27.388Z level=INFO msg="Uploading 1 narinfos" lb # [ 245.834003] niks3-server[1141]: 2026/09/23 04:46:27 INFO Signed narinfos id=88 count=1 lb # [ 245.837806] niks3-server[1141]: 2026/09/23 04:46:27 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst lb # [ 245.862240] niks3-server[1141]: 2026/09/23 04:46:27 INFO Received complete upload request method=POST path=/api/pending_closures/88/complete lb # [ 245.868578] niks3-server[1141]: 2026/09/23 04:46:27 INFO Completed upload id=88 worker2 # [ 245.911696] nix-grpc-daemon[2062]: time=2026-09-23T04:46:27.430Z level=INFO msg="Upload complete. (280ms)" lb # [ 245.873056] niks3-server[1141]: 2026/09/23 04:46:27 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo worker2 # [ 245.924292] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34810 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 245.940322] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 worker2 # [ 245.953619] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) worker2 # [ 245.957850] nix-daemon[843]: reaped child process 2159, status = succeeded client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051... worker2 # [ 245.972612] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) client # farm-a> worker2-next: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 246.609472] nix-daemon[843]: reaped child process 2168, status = succeeded client # farm-a> LOG-farm-a-drain worker2 # [ 246.698834] nix-daemon[843]: accepted connection from pid 2184, user nix-grpc-daemon (trusted) worker2 # [ 246.719925] nix-daemon[843]: reaped child process 2187, status = succeeded lb # [ 246.679712] niks3-server[1141]: 2026/09/23 04:46:28 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 246.737360] nix-grpc-daemon[2062]: time=2026-09-23T04:46:28.254Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 246.738847] nix-grpc-daemon[2062]: time=2026-09-23T04:46:28.254Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)" lb # [ 246.729323] niks3-server[1141]: 2026/09/23 04:46:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/89/sign worker2 # [ 246.779238] nix-grpc-daemon[2062]: time=2026-09-23T04:46:28.297Z level=INFO msg="Uploading 1 narinfos" lb # [ 246.740560] niks3-server[1141]: 2026/09/23 04:46:28 INFO Signed narinfos id=89 count=1 lb # [ 246.742588] niks3-server[1141]: 2026/09/23 04:46:28 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst lb # [ 246.753959] niks3-server[1141]: 2026/09/23 04:46:28 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv lb # [ 246.758068] niks3-server[1141]: 2026/09/23 04:46:28 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls lb # [ 246.764809] niks3-server[1141]: 2026/09/23 04:46:28 INFO Received complete upload request method=POST path=/api/pending_closures/89/complete lb # [ 246.769762] niks3-server[1141]: 2026/09/23 04:46:28 INFO Completed upload id=89 worker2 # [ 246.812851] nix-grpc-daemon[2062]: time=2026-09-23T04:46:28.331Z level=INFO msg="Upload complete. (226ms)" lb # [ 246.771966] niks3-server[1141]: 2026/09/23 04:46:28 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo worker1 # [ 246.817245] nix-grpc-daemon[1475]: ts=2026-09-23T04:46:28Z level=debug event=want client=8 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=0 worker2 # [ 246.825085] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 duration_s=0 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=20 outputs=1 worker1 # [ 246.822997] nix-grpc-daemon[1475]: ts=2026-09-23T04:46:28Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 246.841483] nix-daemon[843]: reaped child process 2166, status = succeeded worker2 # [ 246.872943] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 worker2 # [ 246.888137] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:34810 worker2 # [ 246.902830] nix-grpc-daemon[2062]: time=2026-09-23T04:46:28.420Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'... worker2 # [ 246.913516] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34810 worker2 # [ 246.928281] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) worker2 # [ 247.082820] nix-daemon[843]: accepted connection from pid 2191, user nix-grpc-daemon (trusted) worker2 # [ 247.103412] nix-daemon[843]: reaped child process 2194, status = succeeded lb # [ 247.062349] niks3-server[1141]: 2026/09/23 04:46:28 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 247.115961] nix-grpc-daemon[2062]: time=2026-09-23T04:46:28.634Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 247.117700] nix-grpc-daemon[2062]: time=2026-09-23T04:46:28.634Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)" lb # [ 247.091415] niks3-server[1141]: 2026/09/23 04:46:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/90/sign worker2 # [ 247.141732] nix-grpc-daemon[2062]: time=2026-09-23T04:46:28.660Z level=INFO msg="Uploading 1 narinfos" lb # [ 247.104335] niks3-server[1141]: 2026/09/23 04:46:28 INFO Signed narinfos id=90 count=1 lb # [ 247.107227] niks3-server[1141]: 2026/09/23 04:46:28 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst worker2 # [ 247.167501] nix-grpc-daemon[2062]: time=2026-09-23T04:46:28.686Z level=INFO msg="Upload complete. (222ms)" lb # [ 247.124950] niks3-server[1141]: 2026/09/23 04:46:28 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls lb # [ 247.132114] niks3-server[1141]: 2026/09/23 04:46:28 INFO Received complete upload request method=POST path=/api/pending_closures/90/complete worker2 # [ 247.175261] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34810 duration_s=0 paths=1 nar_bytes_in=632 lb # [ 247.135972] niks3-server[1141]: 2026/09/23 04:46:28 INFO Completed upload id=90 lb # [ 247.139889] niks3-server[1141]: 2026/09/23 04:46:28 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo worker2 # [ 247.189144] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 worker2 # [ 247.194704] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) worker2 # [ 247.199314] nix-daemon[843]: reaped child process 2189, status = succeeded client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051... worker2 # [ 247.209732] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) client # farm-b> worker2-next: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-drain worker2 # [ 248.491050] nix-daemon[843]: reaped child process 2198, status = succeeded worker2 # [ 248.593362] nix-daemon[843]: accepted connection from pid 2214, user nix-grpc-daemon (trusted) worker2 # [ 248.615723] nix-daemon[843]: reaped child process 2217, status = succeeded lb # [ 248.571058] niks3-server[1141]: 2026/09/23 04:46:30 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 248.627373] nix-grpc-daemon[2062]: time=2026-09-23T04:46:30.143Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 248.629007] nix-grpc-daemon[2062]: time=2026-09-23T04:46:30.143Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)" lb # [ 248.613358] niks3-server[1141]: 2026/09/23 04:46:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/91/sign worker2 # [ 248.666983] nix-grpc-daemon[2062]: time=2026-09-23T04:46:30.185Z level=INFO msg="Uploading 1 narinfos" lb # [ 248.627167] niks3-server[1141]: 2026/09/23 04:46:30 INFO Signed narinfos id=91 count=1 lb # [ 248.636644] niks3-server[1141]: 2026/09/23 04:46:30 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst lb # [ 248.647121] niks3-server[1141]: 2026/09/23 04:46:30 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv lb # [ 248.653443] niks3-server[1141]: 2026/09/23 04:46:30 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls worker2 # [ 248.707876] nix-grpc-daemon[2062]: time=2026-09-23T04:46:30.226Z level=INFO msg="Upload complete. (285ms)" lb # [ 248.663649] niks3-server[1141]: 2026/09/23 04:46:30 INFO Received complete upload request method=POST path=/api/pending_closures/91/complete lb # [ 248.669029] niks3-server[1141]: 2026/09/23 04:46:30 INFO Completed upload id=91 lb # [ 248.672708] niks3-server[1141]: 2026/09/23 04:46:30 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo worker2 # [ 248.720772] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=21 outputs=1 worker1 # [ 248.718918] nix-grpc-daemon[1475]: ts=2026-09-23T04:46:30Z level=debug event=want client=8 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=0 worker1 # [ 248.725540] nix-grpc-daemon[1475]: ts=2026-09-23T04:46:30Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 248.742903] nix-daemon[843]: reaped child process 2196, status = succeeded worker2 # [ 248.776950] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 worker2 # [ 248.793160] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:34810 worker2 # [ 248.808578] nix-grpc-daemon[2062]: time=2026-09-23T04:46:30.326Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'... worker2 # [ 248.819119] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34810 worker2 # [ 248.833045] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) worker2 # [ 248.986958] nix-daemon[843]: accepted connection from pid 2221, user nix-grpc-daemon (trusted) worker2 # [ 249.005467] nix-daemon[843]: reaped child process 2224, status = succeeded lb # [ 248.966445] niks3-server[1141]: 2026/09/23 04:46:30 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 249.019762] nix-grpc-daemon[2062]: time=2026-09-23T04:46:30.538Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 249.021630] nix-grpc-daemon[2062]: time=2026-09-23T04:46:30.538Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)" lb # [ 248.995095] niks3-server[1141]: 2026/09/23 04:46:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/92/sign worker2 # [ 249.048417] nix-grpc-daemon[2062]: time=2026-09-23T04:46:30.566Z level=INFO msg="Uploading 1 narinfos" lb # [ 249.010584] niks3-server[1141]: 2026/09/23 04:46:30 INFO Signed narinfos id=92 count=1 lb # [ 249.017107] niks3-server[1141]: 2026/09/23 04:46:30 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst lb # [ 249.024908] niks3-server[1141]: 2026/09/23 04:46:30 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls lb # [ 249.034411] niks3-server[1141]: 2026/09/23 04:46:30 INFO Received complete upload request method=POST path=/api/pending_closures/92/complete lb # [ 249.041077] niks3-server[1141]: 2026/09/23 04:46:30 INFO Completed upload id=92 worker2 # [ 249.083975] nix-grpc-daemon[2062]: time=2026-09-23T04:46:30.602Z level=INFO msg="Upload complete. (233ms)" lb # [ 249.047840] niks3-server[1141]: 2026/09/23 04:46:30 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo worker2 # [ 249.093421] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34810 duration_s=0 paths=1 nar_bytes_in=760 worker2 # [ 249.108292] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 worker2 # [ 249.117695] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) worker2 # [ 249.120324] nix-daemon[843]: reaped child process 2219, status = succeeded client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051... worker2 # [ 249.133834] nix-daemon[843]: accepted connection from pid 2001, user nix-grpc-daemon (trusted) client # farm-top> worker2-next: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-drain worker2 # [ 250.220764] nix-daemon[843]: reaped child process 2228, status = succeeded lb # [ 250.192691] envoy[891]: conn peer=[2001:db8:1::1]:54708 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=931 tx=613 ms=30077 worker2 # [ 250.311312] nix-daemon[843]: accepted connection from pid 2244, user nix-grpc-daemon (trusted) worker2 # [ 250.333825] nix-daemon[843]: reaped child process 2247, status = succeeded lb # [ 250.287780] niks3-server[1141]: 2026/09/23 04:46:31 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 250.344407] nix-grpc-daemon[2062]: time=2026-09-23T04:46:31.861Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 250.345949] nix-grpc-daemon[2062]: time=2026-09-23T04:46:31.861Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)" lb # [ 250.327202] niks3-server[1141]: 2026/09/23 04:46:31 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv lb # [ 250.347134] niks3-server[1141]: 2026/09/23 04:46:31 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst worker2 # [ 250.403871] nix-grpc-daemon[2062]: time=2026-09-23T04:46:31.922Z level=INFO msg="Uploading 1 narinfos" lb # [ 250.361517] niks3-server[1141]: 2026/09/23 04:46:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/93/sign lb # [ 250.371372] niks3-server[1141]: 2026/09/23 04:46:31 INFO Signed narinfos id=93 count=1 lb # [ 250.378328] niks3-server[1141]: 2026/09/23 04:46:31 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls lb # [ 250.388544] niks3-server[1141]: 2026/09/23 04:46:31 INFO Received complete upload request method=POST path=/api/pending_closures/93/complete lb # [ 250.395672] niks3-server[1141]: 2026/09/23 04:46:31 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo lb # [ 250.404209] niks3-server[1141]: 2026/09/23 04:46:31 INFO Completed upload id=93 worker2 # [ 250.448352] nix-grpc-daemon[2062]: time=2026-09-23T04:46:31.966Z level=INFO msg="Upload complete. (304ms)" worker2 # [ 250.461192] nix-grpc-daemon[2001]: ts=2026-09-23T04:46:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34810 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=22 outputs=1 worker2 # [ 250.479260] nix-daemon[843]: reaped child process 2226, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 5.20 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.18 seconds) worker1: must succeed: rm /nix/.rw-store/fill worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.15 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free' worker1 # [ 251.105322] nix-grpc-daemon[1475]: ts=2026-09-23T04:46:32Z level=info event=healthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.10 seconds) (finished: subtest: low disk drains a worker and builds go to the other, in 7.90 seconds) (finished: run the VM test script, in 252.99 seconds) test script finished in 253.23s cleanup kill QemuMachine (pid 50) client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) client # [2026-09-23T04:46:33Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-23T04:46:33Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-23T04:46:33Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 49) lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) lb # [2026-09-23T04:46:34Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-23T04:46:34Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-23T04:46:34Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 51) worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker1 # [2026-09-23T04:46:34Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-23T04:46:34Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-23T04:46:34Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 52) worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker2 # [2026-09-23T04:46:34Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-23T04:46:34Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-23T04:46:34Z INFO virtiofsd] Client disconnected, shutting down (finished: cleanup, in 1.61 seconds) additionally exposed symbols: client, lb, worker1, worker2, vlan1, start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh