tribuchet: building on eliza Machine state will be reset. To keep it, pass --keep-machine-state start all VLans (finished: start all VLans, in 0.00 seconds) Test will time out and terminate in 900.0 seconds run the VM test script start all VMs client: starting vm lb: starting vm worker1: starting vm worker2: starting vm worker2 # Disk image does not exist, creating the virtualisation disk image... client # Disk image does not exist, creating the virtualisation disk image... worker2: QEMU running (pid 52) worker2 # Formatting '/build/vm-state-worker2/tmp.A6jrz2Hj3M', fmt=raw size=1073741824 worker1 # Disk image does not exist, creating the virtualisation disk image... client: QEMU running (pid 49) lb: QEMU running (pid 50) worker2 # mke2fs 1.47.4 (6-Mar-2025) worker1 # Formatting '/build/vm-state-worker1/tmp.iHodXA3zOw', fmt=raw size=1073741824 client # Formatting '/build/vm-state-client/tmp.N1k3HOkeLj', fmt=raw size=1073741824 worker2 # Discarding device blocks: 0/262144 done worker1 # mke2fs 1.47.4 (6-Mar-2025) client # mke2fs 1.47.4 (6-Mar-2025) worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes worker1 # Discarding device blocks: 0/262144 done client # Discarding device blocks: 0/262144 done worker2 # Filesystem UUID: 849c7a3f-1ab2-4693-b52d-af9fe158a12b worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes client # Creating filesystem with 262144 4k blocks and 65536 inodes worker2 # Superblock backups stored on blocks: worker1 # Filesystem UUID: 534bdfd8-5116-405d-9f63-cc51d12f4ec3 client # Filesystem UUID: 1fc68248-1ae2-42bc-8827-5f4f41b076d3 worker2 # 32768, 98304, 163840, 229376 worker1 # Superblock backups stored on blocks: client # Superblock backups stored on blocks: worker2 # worker1 # 32768, 98304, 163840, 229376 client # 32768, 98304, 163840, 229376 worker2 # Allocating group tables: 0/8 done worker1 # client # worker2 # Writing inode tables: 0/8 done worker1 # Allocating group tables: 0/8 done client # Allocating group tables: 0/8 done worker2 # Creating journal (8192 blocks): done worker1 # Writing inode tables: 0/8 done client # Writing inode tables: 0/8 done worker2 # Writing superblocks and filesystem accounting information: 0/8 done worker1 # Creating journal (8192 blocks): done client # Creating journal (8192 blocks): done worker2 # worker1 # Writing superblocks and filesystem accounting information: 0/8 done client # Writing superblocks and filesystem accounting information: 0/8 done worker2 # Virtualisation disk image created. worker1 # client # worker2 # Starting virtiofs daemons... worker1 # Virtualisation disk image created. client # Virtualisation disk image created. worker2 # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # Starting virtiofs daemons... client # Starting virtiofs daemons... worker2 # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-23T04:11:14Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-23T04:11:14Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-23T04:11:14Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-23T04:11:14Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-23T04:11:14Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-23T04:11:14Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-23T04:11:14Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-23T04:11:14Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T04:11:14Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-23T04:11:14Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-23T04:11:14Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T04:11:14Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-23T04:11:14Z INFO virtiofsd] Client connected, servicing requests worker2 # [2026-09-23T04:11:14Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T04:11:14Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-23T04:11:14Z INFO virtiofsd] Client connected, servicing requests lb # Disk image does not exist, creating the virtualisation disk image... worker1: QEMU running (pid 51) worker1 # [2026-09-23T04:11:14Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-23T04:11:14Z INFO virtiofsd] Client connected, servicing requests lb # Formatting '/build/vm-state-lb/tmp.KtLnoNLIN1', fmt=raw size=1073741824 lb # mke2fs 1.47.4 (6-Mar-2025) lb # Discarding device blocks: 0/262144 done lb # Creating filesystem with 262144 4k blocks and 65536 inodes (finished: start all VMs, in 0.49 seconds) lb # Filesystem UUID: 8ad90e9a-c614-4d36-9ae9-6371c810d3cf client: waiting for unit network-addresses-eth1.service lb # Superblock backups stored on blocks: client: waiting for the VM to finish booting lb # 32768, 98304, 163840, 229376 lb # lb # Allocating group tables: 0/8 done lb # Writing inode tables: 0/8 done lb # Creating journal (8192 blocks): done lb # Writing superblocks and filesystem accounting information: 0/8 done lb # lb # Virtualisation disk image created. lb # Starting virtiofs daemons... lb # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-23T04:11:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-23T04:11:14Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-23T04:11:14Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-23T04:11:14Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-23T04:11:14Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-23T04:11:14Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-23T04:11:14Z INFO virtiofsd] Client connected, servicing requests client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 client # [ 0.000000] KASLR enabled client # [ 0.000000] random: crng init done lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] client # [ 0.000000] Machine model: linux,dummy-virt client # [ 0.000000] efi: UEFI not found. worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT lb # [ 0.000000] KASLR enabled worker2 # [ 0.000000] KASLR enabled lb # [ 0.000000] random: crng init done worker2 # [ 0.000000] random: crng init done client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 lb # [ 0.000000] Machine model: linux,dummy-virt worker2 # [ 0.000000] Machine model: linux,dummy-virt worker1 # [ 0.000000] KASLR enabled lb # [ 0.000000] efi: UEFI not found. client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker2 # [ 0.000000] efi: UEFI not found. worker1 # [ 0.000000] random: crng init done client # [ 0.000000] Zone ranges: worker1 # [ 0.000000] Machine model: linux,dummy-virt lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker1 # [ 0.000000] efi: UEFI not found. client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] DMA32 empty lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] Normal empty worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT client # [ 0.000000] Device empty lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] client # [ 0.000000] Movable zone start for each node lb # [ 0.000000] Zone ranges: worker2 # [ 0.000000] Zone ranges: worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] Early memory node ranges lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] lb # [ 0.000000] DMA32 empty worker1 # [ 0.000000] Zone ranges: client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] DMA32 empty lb # [ 0.000000] Normal empty worker2 # [ 0.000000] Normal empty worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] Device empty worker2 # [ 0.000000] Device empty client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] DMA32 empty lb # [ 0.000000] Movable zone start for each node worker2 # [ 0.000000] Movable zone start for each node worker1 # [ 0.000000] Normal empty client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 lb # [ 0.000000] Early memory node ranges worker1 # [ 0.000000] Device empty worker2 # [ 0.000000] Early memory node ranges client # [ 0.000000] psci: probing for conduit method from DT. worker1 # [ 0.000000] Movable zone start for each node lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker1 # [ 0.000000] Early memory node ranges lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] psci: Trusted OS migration not required lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] psci: SMC Calling Convention v1.1 lb # [ 0.000000] psci: probing for conduit method from DT. worker2 # [ 0.000000] psci: probing for conduit method from DT. worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 lb # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware. client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.000000] psci: probing for conduit method from DT. lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware. client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 lb # [ 0.000000] psci: Trusted OS migration not required worker2 # [ 0.000000] psci: Trusted OS migration not required worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs client # [ 0.000000] Detected PIPT I-cache on CPU0 lb # [ 0.000000] psci: SMC Calling Convention v1.1 worker2 # [ 0.000000] psci: SMC Calling Convention v1.1 worker1 # [ 0.000000] psci: Trusted OS migration not required client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker1 # [ 0.000000] psci: SMC Calling Convention v1.1 lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) client # [ 0.000000] CPU features: detected: GICv3 CPU interface lb # [ 0.000000] Detected PIPT I-cache on CPU0 worker2 # [ 0.000000] Detected PIPT I-cache on CPU0 client # [ 0.000000] CPU features: detected: Spectre-v4 worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 client # [ 0.000000] CPU features: detected: Spectre-BHB worker1 # [ 0.000000] Detected PIPT I-cache on CPU0 lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface lb # [ 0.000000] CPU features: detected: GICv3 CPU interface client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface worker2 # [ 0.000000] CPU features: detected: Spectre-v4 client # [ 0.000000] alternatives: applying boot alternatives lb # [ 0.000000] CPU features: detected: Spectre-v4 worker1 # [ 0.000000] CPU features: detected: Spectre-v4 worker2 # [ 0.000000] CPU features: detected: Spectre-BHB lb # [ 0.000000] CPU features: detected: Spectre-BHB worker1 # [ 0.000000] CPU features: detected: Spectre-BHB worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker2 # [ 0.000000] alternatives: applying boot alternatives lb # [ 0.000000] alternatives: applying boot alternatives worker1 # [ 0.000000] alternatives: applying boot alternatives client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/pka8rsnbb5rl489pq588cv7y1qycbh45-nixos-system-client-test/init regInfo=/nix/store/gmykddx7xl27s6hilr51wva292wc8vv2-closure-info/registration console=ttyAMA0,115200n8 console=tty0 client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/gmykddx7xl27s6hilr51wva292wc8vv2-closure-info/registration", will be passed to user space. worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/gnimypw801f4lzy6a5zb7cjxr1n5hqxf-nixos-system-worker2-test/init regInfo=/nix/store/gpp1yfkwxlp6j7wbj4qdvyv3hsy2wnhd-closure-info/registration console=ttyAMA0,115200n8 console=tty0 lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/kgjzg78npi7z07am2szqxkr41f06jp5w-nixos-system-lb-test/init regInfo=/nix/store/lcw0xq38k7svc2ifmp8wza1s56mbyq61-closure-info/registration console=ttyAMA0,115200n8 console=tty0 client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/i9c4dcq9i981i967gfs22mhmyydrkc8s-nixos-system-worker1-test/init regInfo=/nix/store/q0yhvq411jnv0dm976pqpl3xd1a1rg1q-closure-info/registration console=ttyAMA0,115200n8 console=tty0 client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/gpp1yfkwxlp6j7wbj4qdvyv3hsy2wnhd-closure-info/registration", will be passed to user space. lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/lcw0xq38k7svc2ifmp8wza1s56mbyq61-closure-info/registration", will be passed to user space. client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/q0yhvq411jnv0dm976pqpl3xd1a1rg1q-closure-info/registration", will be passed to user space. worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB client # [ 0.000000] software IO TLB: area num 1. worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB client # [ 0.000000] Fallback order for Node 0: 0 worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker2 # [ 0.000000] software IO TLB: area num 1. lb # [ 0.000000] software IO TLB: area num 1. worker1 # [ 0.000000] software IO TLB: area num 1. client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB client # [ 0.000000] Policy zone: DMA lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker2 # [ 0.000000] Fallback order for Node 0: 0 lb # [ 0.000000] Fallback order for Node 0: 0 worker1 # [ 0.000000] Fallback order for Node 0: 0 client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 client # [ 0.000000] allocated 3145728 bytes of page_ext lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker2 # [ 0.000000] Policy zone: DMA lb # [ 0.000000] Policy zone: DMA worker1 # [ 0.000000] Policy zone: DMA client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off client # [ 0.000000] ftrace: allocated 294 pages with 4 groups lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off client # [ 0.000000] rcu: Hierarchical RCU implementation. worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 client # [ 0.000000] rcu: RCU event tracing is enabled. worker2 # [ 0.000000] allocated 3145728 bytes of page_ext worker1 # [ 0.000000] allocated 3145728 bytes of page_ext lb # [ 0.000000] allocated 3145728 bytes of page_ext worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups client # [ 0.000000] Trampoline variant of Tasks RCU enabled. worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker2 # [ 0.000000] rcu: Hierarchical RCU implementation. client # [ 0.000000] Rude variant of Tasks RCU enabled. worker1 # [ 0.000000] rcu: Hierarchical RCU implementation. lb # [ 0.000000] rcu: Hierarchical RCU implementation. worker2 # [ 0.000000] rcu: RCU event tracing is enabled. client # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.000000] rcu: RCU event tracing is enabled. lb # [ 0.000000] rcu: RCU event tracing is enabled. worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled. worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled. lb # [ 0.000000] Trampoline variant of Tasks RCU enabled. client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 worker2 # [ 0.000000] Rude variant of Tasks RCU enabled. worker1 # [ 0.000000] Rude variant of Tasks RCU enabled. lb # [ 0.000000] Rude variant of Tasks RCU enabled. worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled. client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] Tracing variant of Tasks RCU enabled. worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 client # [ 0.000000] GICv3: 256 SPIs implemented worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.000000] GICv3: 0 Extended SPIs implemented lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.000000] Root IRQ handler: gic_handle_irq worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker2 # [ 0.000000] GICv3: 256 SPIs implemented worker1 # [ 0.000000] GICv3: 256 SPIs implemented lb # [ 0.000000] GICv3: 256 SPIs implemented worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented lb # [ 0.000000] GICv3: 0 Extended SPIs implemented worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq lb # [ 0.000000] Root IRQ handler: gic_handle_irq worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1) client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1) lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a00000 client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). client # [ 0.000028] arm-pv: using stolen time PV lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns client # [ 0.000397] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns client # [ 0.000576] Console: colour dummy device 80x25 worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns client # [ 0.000583] printk: legacy console [tty0] enabled lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker1 # [ 0.000028] arm-pv: using stolen time PV worker2 # [ 0.000029] arm-pv: using stolen time PV lb # [ 0.000029] arm-pv: using stolen time PV client # [ 0.000776] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker2 # [ 0.000409] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker1 # [ 0.000402] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) lb # [ 0.000375] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) client # [ 0.000783] pid_max: default: 32768 minimum: 301 worker2 # [ 0.000588] Console: colour dummy device 80x25 worker1 # [ 0.000594] Console: colour dummy device 80x25 lb # [ 0.000576] Console: colour dummy device 80x25 worker2 # [ 0.000596] printk: legacy console [tty0] enabled worker1 # [ 0.000601] printk: legacy console [tty0] enabled client # [ 0.000863] LSM: initializing lsm=capability,landlock,yama,bpf,ima lb # [ 0.000582] printk: legacy console [tty0] enabled client # [ 0.001001] landlock: Up and running. client # [ 0.001004] Yama: becoming mindful. worker2 # [ 0.000791] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker1 # [ 0.000789] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) client # [ 0.001453] LSM support for eBPF active lb # [ 0.000775] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker2 # [ 0.000798] pid_max: default: 32768 minimum: 301 worker1 # [ 0.000796] pid_max: default: 32768 minimum: 301 lb # [ 0.000781] pid_max: default: 32768 minimum: 301 client # [ 0.001583] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.000869] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker1 # [ 0.000867] LSM: initializing lsm=capability,landlock,yama,bpf,ima lb # [ 0.000855] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker1 # [ 0.000985] landlock: Up and running. worker2 # [ 0.000999] landlock: Up and running. client # [ 0.001620] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.001001] Yama: becoming mindful. worker1 # [ 0.000988] Yama: becoming mindful. lb # [ 0.001011] landlock: Up and running. client # [ 0.003425] rcu: Hierarchical SRCU implementation. lb # [ 0.001014] Yama: becoming mindful. worker1 # [ 0.001431] LSM support for eBPF active worker2 # [ 0.001460] LSM support for eBPF active lb # [ 0.001458] LSM support for eBPF active client # [ 0.003430] rcu: Max phase no-delay instances is 1000. worker2 # [ 0.001615] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker1 # [ 0.001570] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.004661] fsl-mc MSI: its@8080000 domain created lb # [ 0.001593] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.004750] EFI services will not be available. worker1 # [ 0.001602] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.001648] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.004857] smp: Bringing up secondary CPUs ... lb # [ 0.001625] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker1 # [ 0.003359] rcu: Hierarchical SRCU implementation. worker2 # [ 0.003511] rcu: Hierarchical SRCU implementation. client # [ 0.004865] smp: Brought up 1 node, 1 CPU lb # [ 0.003382] rcu: Hierarchical SRCU implementation. worker2 # [ 0.003515] rcu: Max phase no-delay instances is 1000. worker1 # [ 0.003364] rcu: Max phase no-delay instances is 1000. client # [ 0.004868] SMP: Total of 1 processors activated. lb # [ 0.003386] rcu: Max phase no-delay instances is 1000. worker1 # [ 0.004566] fsl-mc MSI: its@8080000 domain created worker2 # [ 0.004770] fsl-mc MSI: its@8080000 domain created client # [ 0.004874] CPU: All CPU(s) started at EL1 lb # [ 0.004599] fsl-mc MSI: its@8080000 domain created worker1 # [ 0.004657] EFI services will not be available. worker2 # [ 0.004861] EFI services will not be available. lb # [ 0.004694] EFI services will not be available. client # [ 0.004884] CPU features: detected: Branch Target Identification worker1 # [ 0.004728] smp: Bringing up secondary CPUs ... worker2 # [ 0.004931] smp: Bringing up secondary CPUs ... worker1 # [ 0.004736] smp: Brought up 1 node, 1 CPU lb # [ 0.004759] smp: Bringing up secondary CPUs ... worker2 # [ 0.004939] smp: Brought up 1 node, 1 CPU client # [ 0.004888] CPU features: detected: ARMv8.4 Translation Table Level lb # [ 0.004767] smp: Brought up 1 node, 1 CPU worker1 # [ 0.004739] SMP: Total of 1 processors activated. worker2 # [ 0.004942] SMP: Total of 1 processors activated. worker1 # [ 0.004744] CPU: All CPU(s) started at EL1 lb # [ 0.004770] SMP: Total of 1 processors activated. worker2 # [ 0.004947] CPU: All CPU(s) started at EL1 client # [ 0.004891] CPU features: detected: Instruction cache invalidation not required for I/D coherence lb # [ 0.004776] CPU: All CPU(s) started at EL1 worker1 # [ 0.004753] CPU features: detected: Branch Target Identification worker2 # [ 0.004957] CPU features: detected: Branch Target Identification client # [ 0.004895] CPU features: detected: Data cache clean to the PoU not required for I/D coherence lb # [ 0.004786] CPU features: detected: Branch Target Identification worker1 # [ 0.004758] CPU features: detected: ARMv8.4 Translation Table Level worker2 # [ 0.004962] CPU features: detected: ARMv8.4 Translation Table Level client # [ 0.004899] CPU features: detected: Common not Private translations lb # [ 0.004790] CPU features: detected: ARMv8.4 Translation Table Level worker1 # [ 0.004761] CPU features: detected: Instruction cache invalidation not required for I/D coherence client # [ 0.004903] CPU features: detected: CRC32 instructions worker2 # [ 0.004965] CPU features: detected: Instruction cache invalidation not required for I/D coherence lb # [ 0.004794] CPU features: detected: Instruction cache invalidation not required for I/D coherence client # [ 0.004905] CPU features: detected: Data cache clean to Point of Deep Persistence worker1 # [ 0.004765] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker2 # [ 0.004969] CPU features: detected: Data cache clean to the PoU not required for I/D coherence lb # [ 0.004797] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker1 # [ 0.004769] CPU features: detected: Common not Private translations client # [ 0.004909] CPU features: detected: Data cache clean to Point of Persistence worker2 # [ 0.004972] CPU features: detected: Common not Private translations worker1 # [ 0.004772] CPU features: detected: CRC32 instructions lb # [ 0.004801] CPU features: detected: Common not Private translations worker2 # [ 0.004976] CPU features: detected: CRC32 instructions client # [ 0.004912] CPU features: detected: Data independent timing control (DIT) lb # [ 0.004804] CPU features: detected: CRC32 instructions client # [ 0.004915] CPU features: detected: E0PD worker1 # [ 0.004775] CPU features: detected: Data cache clean to Point of Deep Persistence worker2 # [ 0.004978] CPU features: detected: Data cache clean to Point of Deep Persistence client # [ 0.004917] CPU features: detected: Enhanced Counter Virtualization lb # [ 0.004807] CPU features: detected: Data cache clean to Point of Deep Persistence worker1 # [ 0.004778] CPU features: detected: Data cache clean to Point of Persistence worker2 # [ 0.004982] CPU features: detected: Data cache clean to Point of Persistence lb # [ 0.004810] CPU features: detected: Data cache clean to Point of Persistence client # [ 0.004921] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) worker1 # [ 0.004781] CPU features: detected: Data independent timing control (DIT) worker2 # [ 0.004985] CPU features: detected: Data independent timing control (DIT) worker1 # [ 0.004784] CPU features: detected: E0PD lb # [ 0.004813] CPU features: detected: Data independent timing control (DIT) client # [ 0.004924] CPU features: detected: Enhanced Virtualization Traps worker2 # [ 0.004988] CPU features: detected: E0PD lb # [ 0.004816] CPU features: detected: E0PD worker1 # [ 0.004787] CPU features: detected: Enhanced Counter Virtualization client # [ 0.004927] CPU features: detected: Fine Grained Traps worker2 # [ 0.004991] CPU features: detected: Enhanced Counter Virtualization lb # [ 0.004818] CPU features: detected: Enhanced Counter Virtualization worker1 # [ 0.004790] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) client # [ 0.004931] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker2 # [ 0.004994] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) worker1 # [ 0.004793] CPU features: detected: Enhanced Virtualization Traps client # [ 0.004936] CPU features: detected: RCpc load-acquire (LDAPR) worker2 # [ 0.004997] CPU features: detected: Enhanced Virtualization Traps worker1 # [ 0.004796] CPU features: detected: Fine Grained Traps client # [ 0.004939] CPU features: detected: LSE atomic instructions worker2 # [ 0.005000] CPU features: detected: Fine Grained Traps client # [ 0.004942] CPU features: detected: Privileged Access Never worker1 # [ 0.004800] CPU features: detected: Generic authentication (architected QARMA5 algorithm) client # [ 0.004945] CPU features: detected: PMUv3 worker2 # [ 0.005004] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker1 # [ 0.004805] CPU features: detected: RCpc load-acquire (LDAPR) client # [ 0.004947] CPU features: detected: RAS Extension Support worker2 # [ 0.005008] CPU features: detected: RCpc load-acquire (LDAPR) worker1 # [ 0.004808] CPU features: detected: LSE atomic instructions client # [ 0.004950] CPU features: detected: RASv1p1 Extension Support worker2 # [ 0.005011] CPU features: detected: LSE atomic instructions worker1 # [ 0.004811] CPU features: detected: Privileged Access Never client # [ 0.004953] CPU features: detected: Random Number Generator worker2 # [ 0.005014] CPU features: detected: Privileged Access Never worker1 # [ 0.004813] CPU features: detected: PMUv3 worker2 # [ 0.005016] CPU features: detected: PMUv3 client # [ 0.004955] CPU features: detected: Speculation barrier (SB) worker1 # [ 0.004816] CPU features: detected: RAS Extension Support worker2 # [ 0.005019] CPU features: detected: RAS Extension Support client # [ 0.004958] CPU features: detected: Stage-2 Force Write-Back worker1 # [ 0.004819] CPU features: detected: RASv1p1 Extension Support worker2 # [ 0.005022] CPU features: detected: RASv1p1 Extension Support worker1 # [ 0.004821] CPU features: detected: Random Number Generator worker2 # [ 0.005025] CPU features: detected: Random Number Generator worker1 # [ 0.004824] CPU features: detected: Speculation barrier (SB) worker1 # [ 0.004827] CPU features: detected: Stage-2 Force Write-Back lb # [ 0.004822] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) worker1 # [ 0.004830] CPU features: detected: TLB range maintenance instructions lb # [ 0.004825] CPU features: detected: Enhanced Virtualization Traps worker1 # [ 0.004834] CPU features: detected: Speculative Store Bypassing Safe (SSBS) lb # [ 0.004828] CPU features: detected: Fine Grained Traps worker1 # [ 0.004871] alternatives: applying system-wide alternatives lb # [ 0.004831] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker1 # [ 0.007771] CPU features: detected: BBM Level 2 without TLB conflict abort lb # [ 0.004836] CPU features: detected: RCpc load-acquire (LDAPR) lb # [ 0.004838] CPU features: detected: LSE atomic instructions worker1 # [ 0.008053] Memory: 1405620K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved) lb # [ 0.004841] CPU features: detected: Privileged Access Never worker1 # [ 0.008409] devtmpfs: initialized lb # [ 0.004844] CPU features: detected: PMUv3 worker1 # [ 0.010257] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) lb # [ 0.004847] CPU features: detected: RAS Extension Support client # [ 0.004961] CPU features: detected: TLB range maintenance instructions lb # [ 0.004849] CPU features: detected: RASv1p1 Extension Support worker1 # [ 0.010280] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). client # [ 0.004966] CPU features: detected: Speculative Store Bypassing Safe (SSBS) lb # [ 0.004852] CPU features: detected: Random Number Generator worker2 # [ 0.005027] CPU features: detected: Speculation barrier (SB) worker1 # [ 0.010445] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL client # [ 0.005004] alternatives: applying system-wide alternatives lb # [ 0.004854] CPU features: detected: Speculation barrier (SB) worker1 # [ 0.010449] 0 pages in range for non-PLT usage worker2 # [ 0.005030] CPU features: detected: Stage-2 Force Write-Back client # [ 0.007883] CPU features: detected: BBM Level 2 without TLB conflict abort worker1 # [ 0.010450] 508272 pages in range for PLT usage lb # [ 0.004857] CPU features: detected: Stage-2 Force Write-Back worker2 # [ 0.005033] CPU features: detected: TLB range maintenance instructions worker1 # [ 0.010544] pinctrl core: initialized pinctrl subsystem lb # [ 0.004860] CPU features: detected: TLB range maintenance instructions worker1 # [ 0.011297] DMI not present or invalid. worker2 # [ 0.005038] CPU features: detected: Speculative Store Bypassing Safe (SSBS) lb # [ 0.004864] CPU features: detected: Speculative Store Bypassing Safe (SSBS) client # [ 0.008075] Memory: 1405648K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved) worker2 # [ 0.005075] alternatives: applying system-wide alternatives worker1 # [ 0.014406] NET: Registered PF_NETLINK/PF_ROUTE protocol family client # [ 0.008461] devtmpfs: initialized lb # [ 0.004902] alternatives: applying system-wide alternatives worker2 # [ 0.007949] CPU features: detected: BBM Level 2 without TLB conflict abort worker1 # [ 0.016584] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations client # [ 0.010294] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) lb # [ 0.007753] CPU features: detected: BBM Level 2 without TLB conflict abort worker1 # [ 0.016772] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations client # [ 0.010316] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker2 # [ 0.008177] Memory: 1405656K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved) worker1 # [ 0.016968] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker2 # [ 0.008551] devtmpfs: initialized client # [ 0.010492] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL lb # [ 0.007945] Memory: 1405628K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved) worker1 # [ 0.016988] audit: initializing netlink subsys (disabled) client # [ 0.010497] 0 pages in range for non-PLT usage lb # [ 0.008334] devtmpfs: initialized worker2 # [ 0.010359] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) client # [ 0.010498] 508272 pages in range for PLT usage worker1 # [ 0.017522] thermal_sys: Registered thermal governor 'fair_share' lb # [ 0.010143] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) client # [ 0.010585] pinctrl core: initialized pinctrl subsystem worker1 # [ 0.017524] thermal_sys: Registered thermal governor 'bang_bang' worker2 # [ 0.010386] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). client # [ 0.011309] DMI not present or invalid. lb # [ 0.010164] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker1 # [ 0.017528] thermal_sys: Registered thermal governor 'step_wise' worker2 # [ 0.010568] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL client # [ 0.014487] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker2 # [ 0.010572] 0 pages in range for non-PLT usage lb # [ 0.010327] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL worker1 # [ 0.017531] thermal_sys: Registered thermal governor 'user_space' lb # [ 0.010332] 0 pages in range for non-PLT usage worker2 # [ 0.010573] 508272 pages in range for PLT usage client # [ 0.016690] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker1 # [ 0.017536] thermal_sys: Registered thermal governor 'power_allocator' lb # [ 0.010333] 508272 pages in range for PLT usage worker2 # [ 0.010680] pinctrl core: initialized pinctrl subsystem client # [ 0.016870] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker2 # [ 0.011405] DMI not present or invalid. worker1 # [ 0.017556] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 lb # [ 0.010422] pinctrl core: initialized pinctrl subsystem lb # [ 0.011164] DMI not present or invalid. worker2 # [ 0.014685] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker1 # [ 0.017566] cpuidle: using governor ladder client # [ 0.017069] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker1 # [ 0.017571] cpuidle: using governor menu lb # [ 0.014398] NET: Registered PF_NETLINK/PF_ROUTE protocol family client # [ 0.017089] audit: initializing netlink subsys (disabled) worker2 # [ 0.016881] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker1 # [ 0.017747] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. lb # [ 0.016565] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations client # [ 0.017689] thermal_sys: Registered thermal governor 'fair_share' worker2 # [ 0.017068] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker1 # [ 0.017760] ASID allocator initialised with 65536 entries client # [ 0.017691] thermal_sys: Registered thermal governor 'bang_bang' worker1 # [ 0.018932] Serial: AMBA PL011 UART driver lb # [ 0.016748] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker2 # [ 0.017267] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations client # [ 0.017695] thermal_sys: Registered thermal governor 'step_wise' lb # [ 0.016946] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker2 # [ 0.017287] audit: initializing netlink subsys (disabled) client # [ 0.017698] thermal_sys: Registered thermal governor 'user_space' lb # [ 0.016966] audit: initializing netlink subsys (disabled) worker1 # [ 0.024006] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 worker2 # [ 0.017891] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.017704] thermal_sys: Registered thermal governor 'power_allocator' worker1 # [ 0.024175] printk: console [ttyAMA0] enabled lb # [ 0.017501] thermal_sys: Registered thermal governor 'fair_share' worker2 # [ 0.017893] thermal_sys: Registered thermal governor 'bang_bang' client # [ 0.017728] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 lb # [ 0.017504] thermal_sys: Registered thermal governor 'bang_bang' worker2 # [ 0.017896] thermal_sys: Registered thermal governor 'step_wise' client # [ 0.017738] cpuidle: using governor ladder lb # [ 0.017507] thermal_sys: Registered thermal governor 'step_wise' worker2 # [ 0.017899] thermal_sys: Registered thermal governor 'user_space' client # [ 0.017743] cpuidle: using governor menu lb # [ 0.017510] thermal_sys: Registered thermal governor 'user_space' worker2 # [ 0.017904] thermal_sys: Registered thermal governor 'power_allocator' client # [ 0.017940] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. lb # [ 0.017515] thermal_sys: Registered thermal governor 'power_allocator' client # [ 0.017956] ASID allocator initialised with 65536 entries worker2 # [ 0.017929] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 client # [ 0.019144] Serial: AMBA PL011 UART driver worker2 # [ 0.017937] cpuidle: using governor ladder lb # [ 0.017537] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 worker2 # [ 0.017943] cpuidle: using governor menu lb # [ 0.017547] cpuidle: using governor ladder client # [ 0.024299] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 lb # [ 0.017552] cpuidle: using governor menu worker2 # [ 0.018122] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. client # [ 0.024433] printk: console [ttyAMA0] enabled worker2 # [ 0.018135] ASID allocator initialised with 65536 entries lb # [ 0.017733] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker2 # [ 0.019363] Serial: AMBA PL011 UART driver lb # [ 0.017746] ASID allocator initialised with 65536 entries client # [ 0.143512] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages lb # [ 0.018934] Serial: AMBA PL011 UART driver worker2 # [ 0.024568] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 worker1 # [ 0.140420] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages client # [ 0.143528] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker2 # [ 0.024723] printk: console [ttyAMA0] enabled lb # [ 0.024172] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 worker1 # [ 0.140435] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page client # [ 0.143532] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages lb # [ 0.024288] printk: console [ttyAMA0] enabled worker1 # [ 0.140440] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages client # [ 0.143536] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page worker2 # [ 0.143811] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker1 # [ 0.140444] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page client # [ 0.143541] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker2 # [ 0.143828] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page lb # [ 0.144257] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker1 # [ 0.140449] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages client # [ 0.143545] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.144272] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker2 # [ 0.143833] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages worker1 # [ 0.140453] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page client # [ 0.143549] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages worker2 # [ 0.143837] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page lb # [ 0.144277] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages worker1 # [ 0.140458] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages client # [ 0.143553] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page lb # [ 0.144282] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page worker2 # [ 0.143841] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker1 # [ 0.140462] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker2 # [ 0.143846] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.144286] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker2 # [ 0.143850] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages lb # [ 0.144290] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page client # [ 0.150707] fbcon: Taking over console worker2 # [ 0.143855] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page lb # [ 0.144294] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages client # [ 0.150721] ACPI: Interpreter disabled. worker1 # [ 0.147557] fbcon: Taking over console lb # [ 0.144298] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker1 # [ 0.147571] ACPI: Interpreter disabled. worker2 # [ 0.151102] fbcon: Taking over console worker2 # [ 0.151114] ACPI: Interpreter disabled. lb # [ 0.151555] fbcon: Taking over console lb # [ 0.151568] ACPI: Interpreter disabled. client # [ 0.159302] iommu: Default domain type: Translated client # [ 0.159312] iommu: DMA domain TLB invalidation policy: strict mode worker2 # [ 0.158140] iommu: Default domain type: Translated worker1 # [ 0.156468] iommu: Default domain type: Translated client # [ 0.161057] SCSI subsystem initialized worker2 # [ 0.158149] iommu: DMA domain TLB invalidation policy: strict mode worker1 # [ 0.156477] iommu: DMA domain TLB invalidation policy: strict mode worker1 # [ 0.156805] SCSI subsystem initialized worker2 # [ 0.159808] SCSI subsystem initialized client # [ 0.161722] usbcore: registered new interface driver usbfs lb # [ 0.160315] iommu: Default domain type: Translated client # [ 0.161747] usbcore: registered new interface driver hub worker1 # [ 0.158763] usbcore: registered new interface driver usbfs worker2 # [ 0.160480] usbcore: registered new interface driver usbfs client # [ 0.161786] usbcore: registered new device driver usb lb # [ 0.160325] iommu: DMA domain TLB invalidation policy: strict mode worker1 # [ 0.158799] usbcore: registered new interface driver hub worker2 # [ 0.160512] usbcore: registered new interface driver hub client # [ 0.162053] pps_core: LinuxPPS API ver. 1 registered worker1 # [ 0.158815] usbcore: registered new device driver usb lb # [ 0.162028] SCSI subsystem initialized worker2 # [ 0.160528] usbcore: registered new device driver usb worker1 # [ 0.159095] pps_core: LinuxPPS API ver. 1 registered worker2 # [ 0.160806] pps_core: LinuxPPS API ver. 1 registered client # [ 0.162060] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti client # [ 0.162070] PTP clock support registered lb # [ 0.162668] usbcore: registered new interface driver usbfs worker1 # [ 0.159101] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti client # [ 0.162122] EDAC MC: Ver: 3.0.0 worker2 # [ 0.160813] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker1 # [ 0.159111] PTP clock support registered lb # [ 0.162691] usbcore: registered new interface driver hub worker2 # [ 0.160822] PTP clock support registered worker1 # [ 0.159157] EDAC MC: Ver: 3.0.0 client # [ 0.166588] scmi_core: SCMI protocol bus registered worker2 # [ 0.160876] EDAC MC: Ver: 3.0.0 lb # [ 0.162720] usbcore: registered new device driver usb lb # [ 0.162984] pps_core: LinuxPPS API ver. 1 registered worker1 # [ 0.163541] scmi_core: SCMI protocol bus registered worker2 # [ 0.165321] scmi_core: SCMI protocol bus registered client # [ 0.167503] FPGA manager framework client # [ 0.168421] vgaarb: loaded lb # [ 0.162990] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker1 # [ 0.164493] FPGA manager framework worker2 # [ 0.166301] FPGA manager framework worker1 # [ 0.165445] vgaarb: loaded lb # [ 0.163000] PTP clock support registered worker2 # [ 0.167233] vgaarb: loaded lb # [ 0.163042] EDAC MC: Ver: 3.0.0 client # [ 0.169045] clocksource: Switched to clocksource arch_sys_counter worker1 # [ 0.166064] clocksource: Switched to clocksource arch_sys_counter worker2 # [ 0.167854] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.167576] scmi_core: SCMI protocol bus registered client # [ 0.169622] VFS: Disk quotas dquot_6.6.0 client # [ 0.169655] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) lb # [ 0.168520] FPGA manager framework lb # [ 0.169478] vgaarb: loaded client # [ 0.171965] netfs: FS-Cache loaded client # [ 0.172082] pnp: PnP ACPI: disabled worker1 # [ 0.169525] VFS: Disk quotas dquot_6.6.0 lb # [ 0.170103] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.170688] VFS: Disk quotas dquot_6.6.0 worker1 # [ 0.169566] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) lb # [ 0.170720] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) worker2 # [ 0.170831] VFS: Disk quotas dquot_6.6.0 lb # [ 0.173132] netfs: FS-Cache loaded worker2 # [ 0.170865] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) lb # [ 0.173239] pnp: PnP ACPI: disabled worker2 # [ 0.174513] netfs: FS-Cache loaded worker1 # [ 0.173115] netfs: FS-Cache loaded worker2 # [ 0.174621] pnp: PnP ACPI: disabled worker1 # [ 0.173217] pnp: PnP ACPI: disabled client # [ 0.177978] NET: Registered PF_INET protocol family worker1 # [ 0.177088] NET: Registered PF_INET protocol family client # [ 0.178249] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker2 # [ 0.178534] NET: Registered PF_INET protocol family worker1 # [ 0.177344] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker2 # [ 0.178789] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) lb # [ 0.179164] NET: Registered PF_INET protocol family lb # [ 0.179462] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) client # [ 0.207094] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) client # [ 0.207130] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) client # [ 0.207156] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) client # [ 0.207227] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) lb # [ 0.208231] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) client # [ 0.207370] TCP: Hash tables configured (established 16384 bind 16384) client # [ 0.207446] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) lb # [ 0.208270] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker1 # [ 0.206173] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) worker2 # [ 0.207998] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) client # [ 0.207479] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.208295] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) worker1 # [ 0.206217] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker2 # [ 0.208040] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) client # [ 0.207514] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.208367] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker1 # [ 0.206241] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) client # [ 0.207648] NET: Registered PF_UNIX/PF_LOCAL protocol family worker2 # [ 0.208065] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) lb # [ 0.208510] TCP: Hash tables configured (established 16384 bind 16384) client # [ 0.207673] NET: Registered PF_XDP protocol family worker1 # [ 0.206314] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker2 # [ 0.208137] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) client # [ 0.207689] PCI: CLS 0 bytes, default 64 lb # [ 0.208584] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker1 # [ 0.206456] TCP: Hash tables configured (established 16384 bind 16384) worker2 # [ 0.208279] TCP: Hash tables configured (established 16384 bind 16384) client # [ 0.207898] Trying to unpack rootfs image as initramfs... lb # [ 0.208616] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.206532] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker2 # [ 0.208356] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) lb # [ 0.208652] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.206565] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) worker2 # [ 0.208389] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.208743] NET: Registered PF_UNIX/PF_LOCAL protocol family worker1 # [ 0.206600] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.208763] NET: Registered PF_XDP protocol family worker2 # [ 0.208424] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.208778] PCI: CLS 0 bytes, default 64 worker1 # [ 0.206719] NET: Registered PF_UNIX/PF_LOCAL protocol family worker2 # [ 0.208521] NET: Registered PF_UNIX/PF_LOCAL protocol family client # [ 0.222575] kvm [1]: HYP mode not available worker1 # [ 0.206757] NET: Registered PF_XDP protocol family lb # [ 0.209010] Trying to unpack rootfs image as initramfs... worker2 # [ 0.208547] NET: Registered PF_XDP protocol family worker1 # [ 0.206773] PCI: CLS 0 bytes, default 64 worker2 # [ 0.208563] PCI: CLS 0 bytes, default 64 worker1 # [ 0.207003] Trying to unpack rootfs image as initramfs... worker2 # [ 0.208824] Trying to unpack rootfs image as initramfs... lb # [ 0.223478] kvm [1]: HYP mode not available worker1 # [ 0.223310] kvm [1]: HYP mode not available worker2 # [ 0.225143] kvm [1]: HYP mode not available client # [ 0.315012] Initialise system trusted keyrings client # [ 0.315740] workingset: timestamp_bits=42 max_order=19 bucket_order=0 client # [ 0.316956] squashfs: version 4.0 (2009/01/31) Phillip Lougher lb # [ 0.314631] Initialise system trusted keyrings lb # [ 0.315384] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker1 # [ 0.314621] Initialise system trusted keyrings worker2 # [ 0.316399] Initialise system trusted keyrings lb # [ 0.316722] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker1 # [ 0.315376] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker2 # [ 0.317124] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker1 # [ 0.316643] squashfs: version 4.0 (2009/01/31) Phillip Lougher lb # [ 0.317475] 9p: Installing v9fs 9p2000 file system support worker2 # [ 0.318358] squashfs: version 4.0 (2009/01/31) Phillip Lougher client # [ 0.317812] 9p: Installing v9fs 9p2000 file system support worker1 # [ 0.317379] 9p: Installing v9fs 9p2000 file system support worker2 # [ 0.319096] 9p: Installing v9fs 9p2000 file system support lb # [ 0.338015] Key type asymmetric registered lb # [ 0.338031] Asymmetric key parser 'x509' registered worker1 # [ 0.337975] Key type asymmetric registered worker2 # [ 0.339779] Key type asymmetric registered lb # [ 0.338088] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker1 # [ 0.337992] Asymmetric key parser 'x509' registered worker2 # [ 0.339796] Asymmetric key parser 'x509' registered worker1 # [ 0.338059] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker2 # [ 0.339851] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) client # [ 0.345845] Key type asymmetric registered client # [ 0.345865] Asymmetric key parser 'x509' registered client # [ 0.345939] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) client # [ 0.347986] io scheduler mq-deadline registered client # [ 0.347996] io scheduler kyber registered lb # [ 0.346157] io scheduler mq-deadline registered lb # [ 0.346171] io scheduler kyber registered worker1 # [ 0.346242] io scheduler mq-deadline registered worker2 # [ 0.348002] io scheduler mq-deadline registered worker1 # [ 0.346257] io scheduler kyber registered worker2 # [ 0.348010] io scheduler kyber registered lb # [ 0.354245] pl061_gpio 9030000.pl061: PL061 GPIO chip registered client # [ 0.357186] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker1 # [ 0.354214] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker2 # [ 0.356002] pl061_gpio 9030000.pl061: PL061 GPIO chip registered lb # [ 0.355621] ledtrig-cpu: registered to indicate activity on CPUs lb # [ 0.356003] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker1 # [ 0.355560] ledtrig-cpu: registered to indicate activity on CPUs worker2 # [ 0.357350] ledtrig-cpu: registered to indicate activity on CPUs lb # [ 0.356021] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker1 # [ 0.355970] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker2 # [ 0.357743] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: lb # [ 0.356033] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker1 # [ 0.355987] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker2 # [ 0.357760] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 lb # [ 0.356041] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker1 # [ 0.355999] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker2 # [ 0.357772] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 client # [ 0.358547] ledtrig-cpu: registered to indicate activity on CPUs lb # [ 0.356060] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker1 # [ 0.356007] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 client # [ 0.358942] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker2 # [ 0.357780] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker1 # [ 0.356028] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits lb # [ 0.356083] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] client # [ 0.358959] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker2 # [ 0.357800] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits lb # [ 0.356171] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker1 # [ 0.356050] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] client # [ 0.358980] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker2 # [ 0.357824] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] lb # [ 0.356180] pci_bus 0000:00: root bus resource [bus 00-ff] worker1 # [ 0.356118] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker2 # [ 0.357907] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 lb # [ 0.356186] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] client # [ 0.358990] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker1 # [ 0.356127] pci_bus 0000:00: root bus resource [bus 00-ff] worker2 # [ 0.357916] pci_bus 0000:00: root bus resource [bus 00-ff] lb # [ 0.356191] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.356133] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] client # [ 0.359010] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker2 # [ 0.357922] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] lb # [ 0.356196] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.356138] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker2 # [ 0.357928] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] client # [ 0.359033] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] lb # [ 0.356278] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.356143] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] client # [ 0.359108] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker2 # [ 0.357933] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] client # [ 0.359118] pci_bus 0000:00: root bus resource [bus 00-ff] lb # [ 0.356725] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.356195] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker2 # [ 0.358024] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint lb # [ 0.356967] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] client # [ 0.359124] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker1 # [ 0.356631] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.358463] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.356984] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.359128] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.356825] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.358662] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.357014] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.356843] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.359134] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker2 # [ 0.358679] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.357030] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.356872] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.359191] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker2 # [ 0.358709] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.356888] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.357473] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker2 # [ 0.358726] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.359636] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.357655] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.357334] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.359832] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.359175] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.357671] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.357514] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] client # [ 0.359848] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.359359] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.357701] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.357530] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.359375] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.359878] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.357559] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.359894] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.359405] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.358027] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint client # [ 0.360352] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.360536] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] client # [ 0.360551] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.360580] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.361033] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint client # [ 0.361235] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] client # [ 0.361251] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.361281] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.361733] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.362214] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] client # [ 0.362231] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.362261] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.362280] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.362743] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint client # [ 0.362943] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.362972] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.382218] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint client # [ 0.363430] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint lb # [ 0.382414] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] worker1 # [ 0.382258] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] client # [ 0.363620] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.382430] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.383923] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker1 # [ 0.382287] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.384122] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] client # [ 0.363649] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.382461] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.382322] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.384139] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.382938] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.364056] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker1 # [ 0.382805] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.384169] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.383120] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] client # [ 0.364239] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker1 # [ 0.382992] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.383136] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.384662] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.383008] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.364484] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.384848] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.383166] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.383037] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.364672] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.384863] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.383185] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.383055] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.364701] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.384893] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.383660] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker1 # [ 0.383513] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker2 # [ 0.384911] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.383848] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.383698] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.385368] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint lb # [ 0.383878] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.383727] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.385557] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.384330] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker1 # [ 0.384210] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker2 # [ 0.385587] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.384517] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.384396] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.386040] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint lb # [ 0.384546] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.384426] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.386228] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.384929] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker1 # [ 0.384810] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker2 # [ 0.386257] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.385108] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker1 # [ 0.384989] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker2 # [ 0.386646] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker1 # [ 0.385236] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.385395] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.386828] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] lb # [ 0.385581] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.385423] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.387114] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.385453] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.385611] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.387306] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.385901] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.386057] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.387336] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.405230] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.386108] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.386255] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.405431] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.387797] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.386285] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.386137] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.405463] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.388003] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.386587] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.386731] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.388033] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.405945] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.386772] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.386916] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.406134] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.388492] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.386802] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.386946] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.406164] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.388680] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.387259] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint lb # [ 0.387415] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker2 # [ 0.388710] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.406630] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker1 # [ 0.387541] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] lb # [ 0.387675] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] client # [ 0.406878] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker1 # [ 0.387559] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.387692] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.389178] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint client # [ 0.406895] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.389464] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] lb # [ 0.387722] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.387588] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.406925] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.389481] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.388041] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.388173] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker2 # [ 0.389511] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.407384] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.388222] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.388354] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] client # [ 0.407568] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.388238] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.388370] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.389967] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.407584] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.388267] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.390151] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.388400] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.407614] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.390166] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.388832] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned lb # [ 0.388966] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned client # [ 0.408186] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker2 # [ 0.390196] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.388843] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned lb # [ 0.388977] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned client # [ 0.408197] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker2 # [ 0.390769] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker1 # [ 0.388848] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned lb # [ 0.388982] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker2 # [ 0.390781] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned client # [ 0.408202] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker1 # [ 0.388891] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned lb # [ 0.389025] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned client # [ 0.408246] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker2 # [ 0.390786] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker1 # [ 0.388937] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned lb # [ 0.389070] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned client # [ 0.408293] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker2 # [ 0.390830] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker1 # [ 0.388983] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned lb # [ 0.389117] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned client # [ 0.408339] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker2 # [ 0.390876] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker1 # [ 0.389028] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned lb # [ 0.389162] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned client # [ 0.408387] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker2 # [ 0.390922] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker1 # [ 0.389074] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned lb # [ 0.389209] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned client # [ 0.408434] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker2 # [ 0.390968] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker1 # [ 0.389119] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned lb # [ 0.389256] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned client # [ 0.408480] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker2 # [ 0.391015] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker1 # [ 0.389165] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned lb # [ 0.389303] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned client # [ 0.408527] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker2 # [ 0.391062] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker1 # [ 0.389209] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned lb # [ 0.389349] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned client # [ 0.408573] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker2 # [ 0.391108] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker1 # [ 0.389254] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned lb # [ 0.389395] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned client # [ 0.408620] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned worker2 # [ 0.391154] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker1 # [ 0.389319] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned lb # [ 0.389468] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned client # [ 0.408688] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned worker1 # [ 0.389364] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker2 # [ 0.391200] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned lb # [ 0.389514] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned client # [ 0.408734] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker1 # [ 0.389385] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker2 # [ 0.391269] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned lb # [ 0.389536] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned client # [ 0.408755] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker1 # [ 0.389406] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker2 # [ 0.391316] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned lb # [ 0.389557] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned client # [ 0.408776] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker1 # [ 0.389428] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned worker2 # [ 0.391337] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned lb # [ 0.389578] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned client # [ 0.408797] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned worker1 # [ 0.389449] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker2 # [ 0.391359] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned lb # [ 0.389600] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned client # [ 0.408818] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker1 # [ 0.389470] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned worker2 # [ 0.391381] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned lb # [ 0.389621] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned client # [ 0.408840] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned worker1 # [ 0.389492] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned worker2 # [ 0.391402] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned lb # [ 0.389643] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned client # [ 0.408861] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned worker1 # [ 0.389515] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned worker2 # [ 0.391424] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned lb # [ 0.389667] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned client # [ 0.408885] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned worker1 # [ 0.389537] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker2 # [ 0.391446] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned lb # [ 0.389689] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned client # [ 0.408907] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker1 # [ 0.389559] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned worker2 # [ 0.391471] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned lb # [ 0.389711] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned client # [ 0.408929] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned worker1 # [ 0.389581] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker2 # [ 0.391493] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned lb # [ 0.389732] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker1 # [ 0.389603] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned client # [ 0.408951] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker2 # [ 0.391515] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned lb # [ 0.389754] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned worker1 # [ 0.389624] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned client # [ 0.408973] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned worker2 # [ 0.391538] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned lb # [ 0.389775] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned worker1 # [ 0.389644] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned client # [ 0.408993] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned worker2 # [ 0.391560] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned lb # [ 0.389797] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned worker1 # [ 0.389665] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned client # [ 0.409014] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned lb # [ 0.389818] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned worker2 # [ 0.391586] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned worker1 # [ 0.389686] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned client # [ 0.409035] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned lb # [ 0.389839] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned worker2 # [ 0.391608] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned worker1 # [ 0.389707] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned worker2 # [ 0.391629] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned lb # [ 0.389859] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned worker1 # [ 0.389727] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned worker2 # [ 0.391650] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned lb # [ 0.389881] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned worker1 # [ 0.389752] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.389906] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] worker2 # [ 0.391672] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned worker1 # [ 0.389761] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] worker2 # [ 0.391693] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned lb # [ 0.389915] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] worker1 # [ 0.389766] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] worker2 # [ 0.391720] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.389920] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] worker2 # [ 0.391729] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] worker2 # [ 0.391735] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] client # [ 0.449097] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned client # [ 0.449129] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned client # [ 0.449152] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned client # [ 0.449180] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] client # [ 0.449190] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] lb # [ 0.450803] pci 0000:00:07.0: enabling device (0000 -> 0002) client # [ 0.449195] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] worker1 # [ 0.450658] pci 0000:00:07.0: enabling device (0000 -> 0002) client # [ 0.450001] pci 0000:00:07.0: enabling device (0000 -> 0002) worker2 # [ 0.452634] pci 0000:00:07.0: enabling device (0000 -> 0002) lb # [ 0.476416] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker1 # [ 0.476035] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker2 # [ 0.477949] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) client # [ 0.483271] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker1 # [ 0.478992] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker1 # [ 0.480919] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) client # [ 0.486078] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) client # [ 0.488030] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker2 # [ 0.484077] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker2 # [ 0.485998] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) client # [ 0.491759] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) lb # [ 0.486289] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) lb # [ 0.489774] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.491073] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) lb # [ 0.491897] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) worker1 # [ 0.493091] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) lb # [ 0.493836] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) worker2 # [ 0.496021] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) worker2 # [ 0.497971] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) client # [ 0.502008] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) client # [ 0.504147] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) lb # [ 0.503969] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) client # [ 0.507489] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) lb # [ 0.505927] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker1 # [ 0.503451] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) worker1 # [ 0.505339] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker1 # [ 0.508744] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) client # [ 0.513366] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) client # [ 0.516133] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) worker2 # [ 0.508010] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) worker2 # [ 0.510159] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) lb # [ 0.516408] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker2 # [ 0.511949] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker2 # [ 0.513706] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) worker1 # [ 0.514962] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) worker2 # [ 0.515501] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker1 # [ 0.518005] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) lb # [ 0.519549] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) lb # [ 0.521343] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) client # [ 0.526131] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) client # [ 0.528556] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker1 # [ 0.528279] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker2 # [ 0.529337] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) lb # [ 0.531603] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) client # [ 0.542554] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled lb # [ 0.540499] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker2 # [ 0.544528] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker1 # [ 0.543321] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker1 # [ 0.545762] msm_serial: driver initialized worker2 # [ 0.547035] msm_serial: driver initialized worker1 # [ 0.545944] SuperH (H)SCI(F) driver initialized worker2 # [ 0.547171] SuperH (H)SCI(F) driver initialized worker1 # [ 0.545997] STM32 USART driver initialized worker2 # [ 0.547224] STM32 USART driver initialized lb # [ 0.547207] msm_serial: driver initialized client # [ 0.550250] msm_serial: driver initialized lb # [ 0.547356] SuperH (H)SCI(F) driver initialized client # [ 0.550411] SuperH (H)SCI(F) driver initialized lb # [ 0.547412] STM32 USART driver initialized client # [ 0.550464] STM32 USART driver initialized lb # [ 0.576850] loop: module loaded client # [ 0.579767] loop: module loaded worker2 # [ 0.577664] loop: module loaded lb # [ 0.577012] virtio_blk virtio2: 1/0/0 default/read/poll queues client # [ 0.579928] virtio_blk virtio2: 1/0/0 default/read/poll queues worker2 # [ 0.577826] virtio_blk virtio2: 1/0/0 default/read/poll queues lb # [ 0.577844] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) client # [ 0.580705] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker2 # [ 0.578572] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker1 # [ 0.578965] loop: module loaded worker1 # [ 0.579147] virtio_blk virtio2: 1/0/0 default/read/poll queues worker1 # [ 0.579930] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) client # [ 0.585580] megasas: 07.734.00.00-rc1 client # [ 0.586382] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] worker2 # [ 0.584379] megasas: 07.734.00.00-rc1 client # [ 0.588476] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.585061] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] client # [ 0.588495] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.590683] megasas: 07.734.00.00-rc1 client # [ 0.598038] Using buffer write method lb # [ 0.591463] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] worker1 # [ 0.590666] megasas: 07.734.00.00-rc1 client # [ 0.598106] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] lb # [ 0.593602] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.591329] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] lb # [ 0.593626] Intel/Sharp Extended Query Table at 0x0031 client # [ 0.600878] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.595116] Using buffer write method worker1 # [ 0.593446] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.596970] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 client # [ 0.600900] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.593471] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.597032] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.595178] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker1 # [ 0.595019] Using buffer write method worker2 # [ 0.598486] Using buffer write method worker1 # [ 0.595098] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker2 # [ 0.598558] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] lb # [ 0.596833] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.596854] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.596726] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.605954] Using buffer write method worker1 # [ 0.596746] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.605983] Concatenating MTD devices: worker2 # [ 0.605008] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.605987] (0): "0.flash" lb # [ 0.605991] (1): "0.flash" worker2 # [ 0.605033] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.606002] into device "0.flash" worker2 # [ 0.606333] Using buffer write method worker2 # [ 0.606359] Concatenating MTD devices: worker2 # [ 0.606363] (0): "0.flash" worker2 # [ 0.606367] (1): "0.flash" worker2 # [ 0.606378] into device "0.flash" client # [ 0.614172] Using buffer write method worker1 # [ 0.610551] Using buffer write method client # [ 0.614207] Concatenating MTD devices: worker1 # [ 0.610579] Concatenating MTD devices: client # [ 0.614212] (0): "0.flash" worker1 # [ 0.610584] (0): "0.flash" client # [ 0.614216] (1): "0.flash" worker1 # [ 0.610588] (1): "0.flash" client # [ 0.614226] into device "0.flash" worker1 # [ 0.610595] into device "0.flash" lb # [ 0.869994] Freeing initrd memory: 26948K client # [ 0.876799] Freeing initrd memory: 26972K lb # [ 0.876563] tun: Universal TUN/TAP device driver, 1.6 client # [ 0.883649] tun: Universal TUN/TAP device driver, 1.6 worker1 # [ 0.880189] Freeing initrd memory: 26976K lb # [ 0.880385] thunder_xcv, ver 1.0 lb # [ 0.880429] thunder_bgx, ver 1.0 lb # [ 0.880452] nicpf, ver 1.0 lb # [ 0.881107] e1000: Intel(R) PRO/1000 Network Driver lb # [ 0.881118] e1000: Copyright (c) 1999-2006 Intel Corporation. lb # [ 0.881142] e1000e: Intel(R) PRO/1000 Network Driver lb # [ 0.881150] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. lb # [ 0.881178] igb: Intel(R) Gigabit Ethernet Network Driver client # [ 0.887579] thunder_xcv, ver 1.0 lb # [ 0.881185] igb: Copyright (c) 2007-2014 Intel Corporation. client # [ 0.887635] thunder_bgx, ver 1.0 client # [ 0.887674] nicpf, ver 1.0 lb # [ 0.881207] igbvf: Intel(R) Gigabit Virtual Function Network Driver client # [ 0.888306] e1000: Intel(R) PRO/1000 Network Driver lb # [ 0.881214] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker1 # [ 0.887062] tun: Universal TUN/TAP device driver, 1.6 client # [ 0.888316] e1000: Copyright (c) 1999-2006 Intel Corporation. lb # [ 0.881367] sky2: driver version 1.30 client # [ 0.888343] e1000e: Intel(R) PRO/1000 Network Driver client # [ 0.888351] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. client # [ 0.888385] igb: Intel(R) Gigabit Ethernet Network Driver client # [ 0.888391] igb: Copyright (c) 2007-2014 Intel Corporation. client # [ 0.888441] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker2 # [ 0.890958] Freeing initrd memory: 26944K client # [ 0.888448] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. lb # [ 0.890380] usbcore: registered new interface driver usb-storage client # [ 0.888598] sky2: driver version 1.30 lb # [ 0.890484] usbcore: registered new interface driver usbserial_generic lb # [ 0.890498] usbserial: USB Serial support registered for generic lb # [ 0.891177] hv_vmbus: registering driver hyperv_keyboard lb # [ 0.892523] ehci-pci 0000:00:07.0: EHCI Host Controller worker1 # [ 0.890882] thunder_xcv, ver 1.0 worker1 # [ 0.890928] thunder_bgx, ver 1.0 lb # [ 0.892548] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 client # [ 0.897832] usbcore: registered new interface driver usb-storage worker1 # [ 0.890952] nicpf, ver 1.0 lb # [ 0.892832] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker1 # [ 0.891507] e1000: Intel(R) PRO/1000 Network Driver client # [ 0.897958] usbcore: registered new interface driver usbserial_generic worker1 # [ 0.891514] e1000: Copyright (c) 1999-2006 Intel Corporation. client # [ 0.897977] usbserial: USB Serial support registered for generic worker1 # [ 0.891538] e1000e: Intel(R) PRO/1000 Network Driver client # [ 0.898641] hv_vmbus: registering driver hyperv_keyboard worker1 # [ 0.891545] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. lb # [ 0.898500] rtc-pl031 9010000.pl031: registered as rtc0 worker1 # [ 0.891575] igb: Intel(R) Gigabit Ethernet Network Driver client # [ 0.902946] ehci-pci 0000:00:07.0: EHCI Host Controller lb # [ 0.898534] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T04:11:15 UTC (1790136675) worker1 # [ 0.891581] igb: Copyright (c) 2007-2014 Intel Corporation. lb # [ 0.898905] i2c_dev: i2c /dev entries driver client # [ 0.902974] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker1 # [ 0.891626] igbvf: Intel(R) Gigabit Virtual Function Network Driver client # [ 0.903252] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker1 # [ 0.891632] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker1 # [ 0.891790] sky2: driver version 1.30 worker1 # [ 0.893437] usbcore: registered new interface driver usb-storage client # [ 0.905882] rtc-pl031 9010000.pl031: registered as rtc0 worker1 # [ 0.893558] usbcore: registered new interface driver usbserial_generic worker1 # [ 0.893581] usbserial: USB Serial support registered for generic client # [ 0.905908] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T04:11:15 UTC (1790136675) worker1 # [ 0.894807] ehci-pci 0000:00:07.0: EHCI Host Controller client # [ 0.906252] i2c_dev: i2c /dev entries driver lb # [ 0.904635] sdhci: Secure Digital Host Controller Interface driver lb # [ 0.904658] sdhci: Copyright(c) Pierre Ossman worker1 # [ 0.894831] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 lb # [ 0.904961] Synopsys Designware Multimedia Card Interface Driver lb # [ 0.905352] sdhci-pltfm: SDHCI platform and OF driver helper worker1 # [ 0.895140] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker1 # [ 0.906390] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 lb # [ 0.908862] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker1 # [ 0.906716] hub 1-0:1.0: USB hub found lb # [ 0.909258] hub 1-0:1.0: USB hub found worker1 # [ 0.906734] hub 1-0:1.0: 6 ports detected worker2 # [ 0.910533] tun: Universal TUN/TAP device driver, 1.6 lb # [ 0.910513] hub 1-0:1.0: 6 ports detected client # [ 0.912014] sdhci: Secure Digital Host Controller Interface driver worker1 # [ 0.909681] hv_vmbus: registering driver hyperv_keyboard client # [ 0.912029] sdhci: Copyright(c) Pierre Ossman client # [ 0.912312] Synopsys Designware Multimedia Card Interface Driver client # [ 0.912711] sdhci-pltfm: SDHCI platform and OF driver helper client # [ 0.913106] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 lb # [ 0.912411] hid: raw HID events driver (C) Jiri Kosina worker1 # [ 0.911399] rtc-pl031 9010000.pl031: registered as rtc0 client # [ 0.916924] hub 1-0:1.0: USB hub found lb # [ 0.912707] usbcore: registered new interface driver usbhid worker1 # [ 0.911429] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T04:11:15 UTC (1790136675) lb # [ 0.912717] usbhid: USB HID core driver client # [ 0.917464] hub 1-0:1.0: 6 ports detected worker2 # [ 0.914376] thunder_xcv, ver 1.0 worker1 # [ 0.911778] i2c_dev: i2c /dev entries driver worker2 # [ 0.914422] thunder_bgx, ver 1.0 worker2 # [ 0.914444] nicpf, ver 1.0 worker2 # [ 0.915081] e1000: Intel(R) PRO/1000 Network Driver worker2 # [ 0.915092] e1000: Copyright (c) 1999-2006 Intel Corporation. worker2 # [ 0.915121] e1000e: Intel(R) PRO/1000 Network Driver lb # [ 0.915740] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available client # [ 0.919703] hid: raw HID events driver (C) Jiri Kosina worker2 # [ 0.915128] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. lb # [ 0.917445] drop_monitor: Initializing network drop monitor service client # [ 0.919989] usbcore: registered new interface driver usbhid worker2 # [ 0.915172] igb: Intel(R) Gigabit Ethernet Network Driver worker1 # [ 0.917274] sdhci: Secure Digital Host Controller Interface driver client # [ 0.920000] usbhid: USB HID core driver lb # [ 0.917632] NET: Registered PF_INET6 protocol family worker1 # [ 0.917289] sdhci: Copyright(c) Pierre Ossman worker2 # [ 0.915181] igb: Copyright (c) 2007-2014 Intel Corporation. worker1 # [ 0.917560] Synopsys Designware Multimedia Card Interface Driver lb # [ 0.920763] Segment Routing with IPv6 worker2 # [ 0.915222] igbvf: Intel(R) Gigabit Virtual Function Network Driver lb # [ 0.920797] In-situ OAM (IOAM) with IPv6 worker1 # [ 0.917953] sdhci-pltfm: SDHCI platform and OF driver helper worker2 # [ 0.915228] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. lb # [ 0.920828] NET: Registered PF_PACKET protocol family worker2 # [ 0.915407] sky2: driver version 1.30 client # [ 0.922991] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available client # [ 0.924574] drop_monitor: Initializing network drop monitor service client # [ 0.924698] NET: Registered PF_INET6 protocol family worker1 # [ 0.922382] hid: raw HID events driver (C) Jiri Kosina client # [ 0.927855] Segment Routing with IPv6 worker1 # [ 0.922664] usbcore: registered new interface driver usbhid client # [ 0.927889] In-situ OAM (IOAM) with IPv6 lb # [ 0.922489] 9pnet: Installing 9P2000 support worker1 # [ 0.922674] usbhid: USB HID core driver client # [ 0.927921] NET: Registered PF_PACKET protocol family lb # [ 0.922538] Key type dns_resolver registered worker2 # [ 0.924448] usbcore: registered new interface driver usb-storage worker2 # [ 0.924559] usbcore: registered new interface driver usbserial_generic worker1 # [ 0.925729] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker2 # [ 0.924576] usbserial: USB Serial support registered for generic worker2 # [ 0.925201] hv_vmbus: registering driver hyperv_keyboard worker2 # [ 0.926536] ehci-pci 0000:00:07.0: EHCI Host Controller lb # [ 0.929558] registered taskstats version 1 worker1 # [ 0.928621] drop_monitor: Initializing network drop monitor service client # [ 0.929627] 9pnet: Installing 9P2000 support worker2 # [ 0.926571] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 lb # [ 0.929704] Loading compiled-in X.509 certificates client # [ 0.929672] Key type dns_resolver registered worker1 # [ 0.928793] NET: Registered PF_INET6 protocol family worker2 # [ 0.926805] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker1 # [ 0.930990] Segment Routing with IPv6 worker1 # [ 0.931021] In-situ OAM (IOAM) with IPv6 worker1 # [ 0.931051] NET: Registered PF_PACKET protocol family worker2 # [ 0.932440] rtc-pl031 9010000.pl031: registered as rtc0 client # [ 0.936882] registered taskstats version 1 worker2 # [ 0.932472] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T04:11:15 UTC (1790136675) worker2 # [ 0.932841] i2c_dev: i2c /dev entries driver worker1 # [ 0.932723] 9pnet: Installing 9P2000 support worker1 # [ 0.932811] Key type dns_resolver registered worker2 # [ 0.936043] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker2 # [ 0.936340] hub 1-0:1.0: USB hub found worker2 # [ 0.936380] hub 1-0:1.0: 6 ports detected client # [ 0.937593] Loading compiled-in X.509 certificates lb # [ 0.938871] Demotion targets for Node 0: null lb # [ 0.938987] Key type .fscrypt registered lb # [ 0.938993] Key type fscrypt-provisioning registered lb # [ 0.939091] ima: No TPM chip found, activating TPM-bypass! lb # [ 0.939111] ima: Allocated hash algorithm: sha1 lb # [ 0.939141] ima: No architecture policies found worker1 # [ 0.939948] registered taskstats version 1 lb # [ 0.943372] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker2 # [ 0.941041] sdhci: Secure Digital Host Controller Interface driver worker1 # [ 0.940104] Loading compiled-in X.509 certificates worker2 # [ 0.941062] sdhci: Copyright(c) Pierre Ossman client # [ 0.946415] Demotion targets for Node 0: null worker2 # [ 0.941323] Synopsys Designware Multimedia Card Interface Driver client # [ 0.946542] Key type .fscrypt registered client # [ 0.946548] Key type fscrypt-provisioning registered worker2 # [ 0.941847] sdhci-pltfm: SDHCI platform and OF driver helper client # [ 0.946651] ima: No TPM chip found, activating TPM-bypass! worker2 # [ 0.943616] hid: raw HID events driver (C) Jiri Kosina client # [ 0.946674] ima: Allocated hash algorithm: sha1 client # [ 0.946695] ima: No architecture policies found worker2 # [ 0.947256] usbcore: registered new interface driver usbhid worker2 # [ 0.947270] usbhid: USB HID core driver client # [ 0.950745] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker1 # [ 0.949341] Demotion targets for Node 0: null worker1 # [ 0.949470] Key type .fscrypt registered worker2 # [ 0.949716] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker1 # [ 0.949476] Key type fscrypt-provisioning registered worker2 # [ 0.951432] drop_monitor: Initializing network drop monitor service worker1 # [ 0.949574] ima: No TPM chip found, activating TPM-bypass! worker2 # [ 0.951642] NET: Registered PF_INET6 protocol family worker1 # [ 0.949593] ima: Allocated hash algorithm: sha1 worker1 # [ 0.949617] ima: No architecture policies found worker2 # [ 0.954870] Segment Routing with IPv6 worker2 # [ 0.954902] In-situ OAM (IOAM) with IPv6 worker1 # [ 0.953851] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker2 # [ 0.954953] NET: Registered PF_PACKET protocol family worker2 # [ 0.956698] 9pnet: Installing 9P2000 support worker2 # [ 0.956757] Key type dns_resolver registered lb # [ 0.961759] clk: Disabling unused clocks lb # [ 0.961782] PM: genpd: Disabling unused power domains lb # [ 0.966680] Freeing unused kernel memory: 4736K lb # [ 0.966895] Run /init as init process worker2 # [ 0.964665] registered taskstats version 1 worker2 # [ 0.964845] Loading compiled-in X.509 certificates client # [ 0.970023] clk: Disabling unused clocks client # [ 0.970049] PM: genpd: Disabling unused power domains client # [ 0.974824] Freeing unused kernel memory: 4736K client # [ 0.975037] Run /init as init process worker2 # [ 0.974577] Demotion targets for Node 0: null worker1 # [ 0.972819] clk: Disabling unused clocks worker2 # [ 0.974696] Key type .fscrypt registered worker1 # [ 0.972844] PM: genpd: Disabling unused power domains worker2 # [ 0.974702] Key type fscrypt-provisioning registered worker2 # [ 0.974820] ima: No TPM chip found, activating TPM-bypass! worker2 # [ 0.974846] ima: Allocated hash algorithm: sha1 worker2 # [ 0.974875] ima: No architecture policies found worker2 # [ 0.979139] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker1 # [ 0.977583] Freeing unused kernel memory: 4736K worker1 # [ 0.977808] Run /init as init process lb # [ 0.984623] systemd[1]: Successfully made /usr/ read-only. client # [ 0.991424] systemd[1]: Successfully made /usr/ read-only. worker1 # [ 0.996435] systemd[1]: Successfully made /usr/ read-only. worker2 # [ 0.998260] clk: Disabling unused clocks worker2 # [ 0.998285] PM: genpd: Disabling unused power domains worker2 # [ 1.002839] Freeing unused kernel memory: 4736K worker2 # [ 1.003075] Run /init as init process worker2 # [ 1.018347] systemd[1]: Successfully made /usr/ read-only. worker1 # [ 1.154186] usb 1-1: new high-speed USB device number 2 using ehci-pci lb # [ 1.162251] usb 1-1: new high-speed USB device number 2 using ehci-pci client # [ 1.165246] usb 1-1: new high-speed USB device number 2 using ehci-pci worker2 # [ 1.183941] usb 1-1: new high-speed USB device number 2 using ehci-pci worker1 # [ 1.309204] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 lb # [ 1.315156] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 client # [ 1.318337] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 lb # [ 1.322257] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 1.326714] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 1.335111] systemd[1]: Detected virtualization qemu. client # [ 1.339514] systemd[1]: Detected virtualization qemu. lb # [ 1.337344] systemd[1]: Detected architecture arm64. lb # [ 1.339481] systemd[1]: Running in initrd. client # [ 1.341748] systemd[1]: Detected architecture arm64. client # [ 1.343731] systemd[1]: Running in initrd. worker2 # [ 1.338758] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker1 # [ 1.331551] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 1.343145] systemd[1]: Initializing machine ID from random generator. worker1 # [ 1.343957] systemd[1]: Detected virtualization qemu. client # [ 1.347126] systemd[1]: Initializing machine ID from random generator. lb # [ 1.346545] systemd[1]: Hostname set to . worker1 # [ 1.346176] systemd[1]: Detected architecture arm64. client # [ 1.350389] systemd[1]: Hostname set to . worker1 # [ 1.348139] systemd[1]: Running in initrd. worker1 # [ 1.351321] systemd[1]: Initializing machine ID from random generator. worker1 # [ 1.354659] systemd[1]: Hostname set to . worker2 # [ 1.353562] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 1.365986] systemd[1]: Detected virtualization qemu. worker2 # [ 1.368343] systemd[1]: Detected architecture arm64. worker2 # [ 1.370363] systemd[1]: Running in initrd. worker2 # [ 1.373554] systemd[1]: Initializing machine ID from random generator. worker2 # [ 1.376618] systemd[1]: Hostname set to . client # [ 1.401641] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker1 # [ 1.398477] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 lb # [ 1.402549] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker2 # [ 1.428440] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 client # [ 1.525118] usb 1-2: new high-speed USB device number 3 using ehci-pci worker1 # [ 1.522175] usb 1-2: new high-speed USB device number 3 using ehci-pci lb # [ 1.526146] usb 1-2: new high-speed USB device number 3 using ehci-pci worker2 # [ 1.552060] usb 1-2: new high-speed USB device number 3 using ehci-pci lb # [ 1.676765] systemd[1]: bpf-restrict-fs: LSM BPF program attached client # [ 1.683784] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker1 # [ 1.680849] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 client # [ 1.693385] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker1 # [ 1.690433] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 lb # [ 1.692074] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 lb # [ 1.698573] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 client # [ 1.707722] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 1.709785] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.715843] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker1 # [ 1.734969] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 1.754113] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 1.796412] systemd[1]: Queued start job for default target Initrd Default Target. client # [ 1.804203] systemd[1]: Queued start job for default target Initrd Default Target. lb # [ 1.804710] systemd[1]: Created slice Slice /system/modprobe. lb # [ 1.805925] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 1.807237] systemd[1]: Expecting device /dev/disk/by-label/nixos... lb # [ 1.808244] systemd[1]: Reached target Path Units. lb # [ 1.808991] systemd[1]: Reached target Slice Units. lb # [ 1.809744] systemd[1]: Reached target Swaps. lb # [ 1.810479] systemd[1]: Reached target Timer Units. lb # [ 1.811446] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 1.812589] systemd[1]: Listening on Journal Socket (/dev/log). client # [ 1.815731] systemd[1]: Created slice Slice /system/modprobe. lb # [ 1.813636] systemd[1]: Listening on Journal Sockets. client # [ 1.816980] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 1.818369] systemd[1]: Expecting device /dev/disk/by-label/nixos... lb # [ 1.814636] systemd[1]: Listening on udev Control Socket. client # [ 1.819400] systemd[1]: Reached target Path Units. lb # [ 1.814799] systemd[1]: Listening on udev Kernel Socket. lb # [ 1.814832] systemd[1]: Reached target Socket Units. client # [ 1.820303] systemd[1]: Reached target Slice Units. client # [ 1.821139] systemd[1]: Reached target Swaps. lb # [ 1.819134] systemd[1]: Starting Create List of Static Device Nodes... client # [ 1.821874] systemd[1]: Reached target Timer Units. client # [ 1.822914] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 1.820221] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 1.824141] systemd[1]: Listening on Journal Socket (/dev/log). client # [ 1.825325] systemd[1]: Listening on Journal Sockets. client # [ 1.825496] systemd[1]: Listening on udev Control Socket. client # [ 1.825656] systemd[1]: Listening on udev Kernel Socket. client # [ 1.825691] systemd[1]: Reached target Socket Units. lb # [ 1.827518] systemd[1]: Mounting Kernel Configuration File System... client # [ 1.830732] systemd[1]: Starting Create List of Static Device Nodes... client # [ 1.831852] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 1.837019] systemd[1]: Queued start job for default target Initrd Default Target. lb # [ 1.839467] systemd[1]: Starting Journal Service... client # [ 1.841326] systemd[1]: Mounting Kernel Configuration File System... client # [ 1.849408] systemd[1]: Starting Journal Service... worker1 # [ 1.849858] systemd[1]: Created slice Slice /system/modprobe. worker2 # [ 1.852706] systemd[1]: Queued start job for default target Initrd Default Target. worker1 # [ 1.851200] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 1.852493] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker1 # [ 1.853516] systemd[1]: Reached target Path Units. worker1 # [ 1.854427] systemd[1]: Reached target Slice Units. worker1 # [ 1.855234] systemd[1]: Reached target Swaps. worker1 # [ 1.855964] systemd[1]: Reached target Timer Units. worker1 # [ 1.856969] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 1.858197] systemd[1]: Listening on Journal Socket (/dev/log). worker1 # [ 1.859329] systemd[1]: Listening on Journal Sockets. lb # [ 1.861186] systemd[1]: Starting Load Kernel Modules... worker1 # [ 1.860451] systemd[1]: Listening on udev Control Socket. worker1 # [ 1.861544] systemd[1]: Listening on udev Kernel Socket. lb # [ 1.862360] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 1.863430] systemd[1]: Created slice Slice /system/modprobe. worker2 # [ 1.864759] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 1.862479] systemd[1]: Reached target Socket Units. worker2 # [ 1.865973] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker2 # [ 1.866947] systemd[1]: Reached target Path Units. worker1 # [ 1.865293] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 1.867728] systemd[1]: Reached target Slice Units. worker2 # [ 1.868559] systemd[1]: Reached target Swaps. worker1 # [ 1.866529] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 1.869239] systemd[1]: Reached target Timer Units. lb # [ 1.867663] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 1.870181] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 1.871311] systemd[1]: Listening on Journal Socket (/dev/log). worker2 # [ 1.872441] systemd[1]: Listening on Journal Sockets. worker2 # [ 1.873346] systemd[1]: Listening on udev Control Socket. worker2 # [ 1.874289] systemd[1]: Listening on udev Kernel Socket. worker2 # [ 1.875224] systemd[1]: Reached target Socket Units. worker2 # [ 1.877838] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 1.878271] systemd[1]: Mounting Kernel Configuration File System... client # [ 1.881289] systemd[1]: Starting Load Kernel Modules... worker2 # [ 1.877921] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 1.881405] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 1.882540] systemd[1]: Starting Journal Service... lb # [ 1.887345] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 1.888089] systemd[1]: Mounting Kernel Configuration File System... client # [ 1.893229] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 1.896156] systemd[1]: Starting Journal Service... client # [ 1.900965] systemd-journald[72]: Collecting audit messages is disabled. worker1 # [ 1.898269] systemd[1]: Starting Load Kernel Modules... worker1 # [ 1.899138] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 1.903468] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 1.904065] systemd[1]: Mounted Kernel Configuration File System. client # [ 1.905870] systemd[1]: Finished Create List of Static Device Nodes. client # [ 1.917327] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 1.917821] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 1.916462] systemd[1]: Starting Coldplug All udev Devices... lb # [ 1.919414] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker2 # [ 1.921211] systemd[1]: Starting Load Kernel Modules... worker2 # [ 1.922090] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 1.923120] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 1.934484] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker2 # [ 1.942566] systemd-journald[72]: Collecting audit messages is disabled. worker2 # [ 1.943605] systemd[1]: Starting Coldplug All udev Devices... lb # [ 1.943689] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 lb # [ 1.943927] [drm] features: -virgl +edid -resource_blob -host_visible lb # [ 1.943935] [drm] features: -context_init lb # [ 1.944699] [drm] number of scanouts: 1 lb # [ 1.944716] [drm] number of cap sets: 0 worker1 # [ 1.946297] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker1 # [ 1.950810] systemd[1]: Mounted Kernel Configuration File System. lb # [ 1.960147] systemd-journald[72]: Collecting audit messages is disabled. client # [ 1.965846] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 1.969429] systemd[1]: Starting Create Static Device Nodes in /dev... lb # [ 1.967052] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 1.967989] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 1.970461] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 1.969131] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 1.970544] systemd-journald[72]: Collecting audit messages is disabled. worker2 # [ 1.980325] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 1.985919] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. lb # [ 1.978606] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic lb # [ 1.978628] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 worker2 # [ 1.989444] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. client # [ 1.997219] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker1 # [ 1.987876] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker1 # [ 1.988764] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 1.990414] systemd[1]: Starting Create Static Device Nodes in /dev... client # [ 2.005026] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker2 # [ 2.003974] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev lb # [ 1.998394] Console: switching to colour frame buffer device 160x50 worker2 # [ 2.013680] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker2 # [ 2.013928] [drm] features: -virgl +edid -resource_blob -host_visible worker2 # [ 2.013940] [drm] features: -context_init worker2 # [ 2.014706] [drm] number of scanouts: 1 worker2 # [ 2.014724] [drm] number of cap sets: 0 worker1 # [ 2.014209] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker1 # [ 2.020645] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 client # [ 2.026444] [drm] features: -virgl +edid -resource_blob -host_visible worker1 # [ 2.020906] [drm] features: -virgl +edid -resource_blob -host_visible client # [ 2.026464] [drm] features: -context_init worker1 # [ 2.020916] [drm] features: -context_init client # [ 2.027253] [drm] number of scanouts: 1 worker1 # [ 2.021652] [drm] number of scanouts: 1 worker1 # [ 2.021669] [drm] number of cap sets: 0 client # [ 2.027271] [drm] number of cap sets: 0 lb # [ 2.026829] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 2.027021] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 2.027048] systemd[1]: Reached target Local File Systems. lb # [ 2.031437] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device client # [ 2.033505] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic client # [ 2.033533] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 lb # [ 2.039425] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker2 # [ 2.040742] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 2.043543] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 2.046750] systemd[1]: Finished Create Static Device Nodes in /dev. worker1 # [ 2.046973] systemd[1]: Reached target Preparation for Local File Systems. worker1 # [ 2.047004] systemd[1]: Reached target Local File Systems. client # [ 2.053002] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 2.053277] systemd[1]: Reached target Preparation for Local File Systems. client # [ 2.053310] systemd[1]: Reached target Local File Systems. worker2 # [ 2.048337] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic lb # [ 2.054287] systemd[1]: Finished Load Kernel Modules. worker2 # [ 2.048359] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 worker1 # [ 2.051184] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker1 # [ 2.051202] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 client # [ 2.061572] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 2.062543] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 2.059540] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 2.069360] Console: switching to colour frame buffer device 160x50 client # [ 2.075901] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker2 # [ 2.072207] Console: switching to colour frame buffer device 160x50 worker2 # [ 2.096591] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device client # [ 2.089618] systemd-modules-load[73]: Inserted module 'dm_mod' client # [ 2.103852] systemd[1]: Finished Load Kernel Modules. client # [ 2.106042] systemd[1]: Started Journal Service. worker1 # [ 2.102132] Console: switching to colour frame buffer device 160x50 worker1 # [ 2.102834] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device client # [ 2.096415] systemd-modules-load[73]: Module 'virtio_balloon' is built in client # [ 2.097696] systemd-modules-load[73]: Module 'virtio_console' is built in client # [ 2.098811] systemd-modules-load[73]: Inserted module 'virtio_gpu' worker2 # [ 2.108597] systemd[1]: Finished Load Kernel Modules. client # [ 2.099883] systemd-modules-load[73]: Module 'virtio_rng' is built in worker2 # [ 2.112868] systemd[1]: Starting Apply Kernel Variables... lb # [ 2.119022] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 2.118882] systemd[1]: Finished Load Kernel Modules. worker2 # [ 2.128326] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 2.128607] systemd[1]: Reached target Preparation for Local File Systems. client # [ 2.120511] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 2.128648] systemd[1]: Reached target Local File Systems. worker1 # [ 2.126505] systemd[1]: Starting Apply Kernel Variables... lb # [ 2.142806] systemd[1]: Started Journal Service. client # [ 2.135915] systemd[1]: Starting Create System Files and Directories... worker2 # [ 2.144331] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 2.140342] systemd-modules-load[73]: Inserted module 'dm_mod' worker2 # [ 2.146653] systemd-modules-load[74]: Inserted module 'dm_mod' lb # [ 2.144263] systemd-modules-load[73]: Module 'virtio_balloon' is built in lb # [ 2.145420] systemd-modules-load[73]: Module 'virtio_console' is built in worker2 # [ 2.159784] systemd[1]: Started Journal Service. lb # [ 2.146498] systemd-modules-load[73]: Inserted module 'virtio_gpu' lb # [ 2.147499] systemd-modules-load[73]: Module 'virtio_rng' is built in worker2 # [ 2.151826] systemd-modules-load[74]: Module 'virtio_balloon' is built in worker2 # [ 2.153334] systemd-modules-load[74]: Module 'virtio_console' is built in worker2 # [ 2.157104] systemd-modules-load[74]: Inserted module 'virtio_gpu' lb # [ 2.157734] systemd[1]: Starting Create System Files and Directories... worker2 # [ 2.164408] systemd-modules-load[74]: Module 'virtio_rng' is built in worker2 # [ 2.168343] systemd[1]: Finished Apply Kernel Variables. lb # [ 2.165724] systemd-udevd[79]: Using default interface naming scheme 'v261'. client # [ 2.171183] systemd-udevd[79]: Using default interface naming scheme 'v261'. worker1 # [ 2.184640] systemd[1]: Started Journal Service. worker2 # [ 2.177241] systemd[1]: Starting Create System Files and Directories... client # [ 2.184353] systemd[1]: Finished Create System Files and Directories. worker1 # [ 2.180366] systemd-modules-load[73]: Inserted module 'dm_mod' worker1 # [ 2.181635] systemd-modules-load[73]: Module 'virtio_balloon' is built in worker1 # [ 2.184313] systemd-modules-load[73]: Module 'virtio_console' is built in lb # [ 2.189031] systemd[1]: Finished Create System Files and Directories. worker1 # [ 2.188396] systemd-modules-load[73]: Inserted module 'virtio_gpu' worker1 # [ 2.190529] systemd-modules-load[73]: Module 'virtio_rng' is built in client # [ 2.197379] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 2.197049] systemd-udevd[79]: Using default interface naming scheme 'v261'. worker1 # [ 2.199550] systemd[1]: Starting Create System Files and Directories... worker1 # [ 2.202316] systemd[1]: Finished Apply Kernel Variables. lb # [ 2.213715] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 2.217029] systemd[1]: Finished Create System Files and Directories. client # [ 2.221992] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 2.230553] systemd-udevd[80]: Using default interface naming scheme 'v261'. worker1 # [ 2.242221] systemd[1]: Finished Create System Files and Directories. worker1 # [ 2.244868] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 2.267259] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 2.278001] systemd[1]: Starting Virtual Console Setup... client # [ 2.288113] systemd[1]: Starting Virtual Console Setup... worker1 # [ 2.304934] systemd[1]: Starting Virtual Console Setup... lb # [ 2.332493] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 2.336860] systemd[1]: Starting Virtual Console Setup... client # [ 2.338087] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 2.335924] systemd[1]: Finished Virtual Console Setup. client # [ 2.339993] systemd[1]: Finished Virtual Console Setup. worker1 # [ 2.373462] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 2.376248] systemd[1]: Finished Virtual Console Setup. worker2 # [ 2.396464] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 2.399996] systemd[1]: Finished Virtual Console Setup. lb # [ 3.025809] systemd[1]: Finished Coldplug All udev Devices. lb # [ 3.026779] systemd[1]: Reached target System Initialization. lb # [ 3.027609] systemd[1]: Reached target Basic System. client # [ 3.039521] systemd[1]: Finished Coldplug All udev Devices. client # [ 3.044087] systemd[1]: Reached target System Initialization. client # [ 3.044984] systemd[1]: Reached target Basic System. worker1 # [ 3.053495] systemd[1]: Finished Coldplug All udev Devices. worker1 # [ 3.054506] systemd[1]: Reached target System Initialization. worker1 # [ 3.055314] systemd[1]: Reached target Basic System. worker2 # [ 3.141206] systemd[1]: Finished Coldplug All udev Devices. worker2 # [ 3.142209] systemd[1]: Reached target System Initialization. worker2 # [ 3.144185] systemd[1]: Reached target Basic System. lb # [ 3.187973] (udev-worker)[92]: Network interface NamePolicy= disabled on kernel command line. client # [ 3.235150] (udev-worker)[92]: Network interface NamePolicy= disabled on kernel command line. lb # [ 3.230897] (udev-worker)[95]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. lb # [ 3.234276] (udev-worker)[95]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.243574] (udev-worker)[97]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 3.249329] (udev-worker)[90]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 3.254731] (udev-worker)[90]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.257873] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.265757] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line. lb # [ 3.308614] systemd[1]: Found device /dev/disk/by-label/nixos. worker2 # [ 3.316064] (udev-worker)[91]: Network interface NamePolicy= disabled on kernel command line. lb # [ 3.315378] systemd[1]: Reached target Initrd Root Device. lb # [ 3.323931] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker1 # [ 3.328803] systemd[1]: Found device /dev/disk/by-label/nixos. worker1 # [ 3.332639] systemd[1]: Reached target Initrd Root Device. worker2 # [ 3.335031] (udev-worker)[90]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 3.335474] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker2 # [ 3.339960] (udev-worker)[90]: Network interface NamePolicy= disabled on kernel command line. client # [ 3.348130] systemd[1]: Found device /dev/disk/by-label/nixos. client # [ 3.353590] systemd[1]: Reached target Initrd Root Device. client # [ 3.357424] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... lb # [ 3.372273] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks lb # [ 3.378680] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. lb # [ 3.391759] systemd[1]: Mounting /sysroot... worker1 # [ 3.391530] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks worker2 # [ 3.399055] systemd[1]: Found device /dev/disk/by-label/nixos. worker2 # [ 3.400810] systemd[1]: Reached target Initrd Root Device. worker1 # [ 3.397157] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. client # [ 3.404102] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks worker2 # [ 3.405104] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... client # [ 3.412189] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker1 # [ 3.408441] systemd[1]: Mounting /sysroot... client # [ 3.416138] systemd[1]: Mounting /sysroot... lb # [ 3.444489] EXT4-fs (vda): mounted filesystem 8ad90e9a-c614-4d36-9ae9-6371c810d3cf r/w with ordered data mode. Quota mode: none. lb # [ 3.436150] systemd[1]: Mounted /sysroot. lb # [ 3.437214] systemd[1]: Reached target Initrd Root File System. lb # [ 3.441463] systemd[1]: Starting Mountpoints Configured in the Real Root... worker1 # [ 3.460990] EXT4-fs (vda): mounted filesystem 534bdfd8-5116-405d-9f63-cc51d12f4ec3 r/w with ordered data mode. Quota mode: none. worker1 # [ 3.454512] systemd[1]: Mounted /sysroot. client # [ 3.472174] EXT4-fs (vda): mounted filesystem 1fc68248-1ae2-42bc-8827-5f4f41b076d3 r/w with ordered data mode. Quota mode: none. worker1 # [ 3.456423] systemd[1]: Reached target Initrd Root File System. client # [ 3.465136] systemd[1]: Mounted /sysroot. worker1 # [ 3.460348] systemd[1]: Starting Mountpoints Configured in the Real Root... worker2 # [ 3.466651] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks client # [ 3.467914] systemd[1]: Reached target Initrd Root File System. client # [ 3.474763] systemd[1]: Starting Mountpoints Configured in the Real Root... worker2 # [ 3.474404] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. lb # [ 3.471738] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. worker2 # [ 3.480969] systemd[1]: Mounting /sysroot... lb # [ 3.480093] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... lb # [ 3.484588] systemd[1]: Reloading... worker1 # [ 3.492820] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. worker1 # [ 3.497607] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... worker1 # [ 3.502472] systemd[1]: Reloading... client # [ 3.513215] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. client # [ 3.519773] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... client # [ 3.521742] systemd[1]: Reloading... worker2 # [ 3.539585] EXT4-fs (vda): mounted filesystem 849c7a3f-1ab2-4693-b52d-af9fe158a12b r/w with ordered data mode. Quota mode: none. worker2 # [ 3.534157] systemd[1]: Mounted /sysroot. worker2 # [ 3.536606] systemd[1]: Reached target Initrd Root File System. worker2 # [ 3.542577] systemd[1]: Starting Mountpoints Configured in the Real Root... worker2 # [ 3.578177] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. worker2 # [ 3.582714] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... worker2 # [ 3.588862] systemd[1]: Reloading... lb # [ 3.717376] systemd[1]: Reloading finished in 238 ms. worker1 # [ 3.748940] systemd[1]: Reloading finished in 247 ms. lb # [ 3.750754] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... lb # [ 3.754068] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... lb # [ 3.762806] systemd[1]: initrd-parse-etc.service: Deactivated successfully. lb # [ 3.764698] systemd[1]: Finished Mountpoints Configured in the Real Root. lb # [ 3.765756] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. client # [ 3.775968] systemd[1]: Reloading finished in 257 ms. worker1 # [ 3.781464] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... worker1 # [ 3.786448] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... worker1 # [ 3.792468] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker1 # [ 3.795235] systemd[1]: Finished Mountpoints Configured in the Real Root. worker1 # [ 3.797454] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. client # [ 3.809429] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... client # [ 3.814720] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... client # [ 3.822639] systemd[1]: initrd-parse-etc.service: Deactivated successfully. client # [ 3.825317] systemd[1]: Finished Mountpoints Configured in the Real Root. client # [ 3.826933] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker2 # [ 3.838691] systemd[1]: Reloading finished in 252 ms. lb # [ 3.860866] systemd[1]: Mounting /sysroot/nix/.ro-store... worker2 # [ 3.873294] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... worker2 # [ 3.878598] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... lb # [ 3.879304] systemd[1]: Mounting /sysroot/nix/.rw-store... worker2 # [ 3.885839] systemd[1]: initrd-parse-etc.service: Deactivated successfully. lb # [ 3.884088] systemd[1]: Mounting /sysroot/run... client # [ 3.887951] systemd[1]: Mounting /sysroot/nix/.ro-store... worker2 # [ 3.888608] systemd[1]: Finished Mountpoints Configured in the Real Root. worker2 # [ 3.890074] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. client # [ 3.898469] systemd[1]: Mounting /sysroot/nix/.rw-store... client # [ 3.910935] systemd[1]: Mounting /sysroot/run... worker1 # [ 3.914395] systemd[1]: Mounting /sysroot/nix/.ro-store... lb # [ 3.924254] systemd[1]: Mounting /sysroot/tmp/shared... client # [ 3.932125] systemd[1]: Mounting /sysroot/tmp/shared... worker1 # [ 3.930916] systemd[1]: Mounting /sysroot/nix/.rw-store... worker1 # [ 3.935053] systemd[1]: Mounting /sysroot/run... worker2 # [ 3.941691] systemd[1]: Mounting /sysroot/nix/.ro-store... lb # [ 3.943857] systemd[1]: Mounting /sysroot/tmp/xchg... lb # [ 3.948110] systemd[1]: Mounted /sysroot/nix/.rw-store. client # [ 3.951874] systemd[1]: Mounting /sysroot/tmp/xchg... worker2 # [ 3.953575] systemd[1]: Mounting /sysroot/nix/.rw-store... worker1 # [ 3.952602] systemd[1]: Mounting /sysroot/tmp/shared... worker2 # [ 3.973288] systemd[1]: Mounting /sysroot/run... worker1 # [ 3.972840] systemd[1]: Mounting /sysroot/tmp/xchg... worker2 # [ 3.982627] systemd[1]: Mounting /sysroot/tmp/shared... lb # [ 3.989828] systemd[1]: Mounted /sysroot/run. client # [ 4.001123] systemd[1]: Mounted /sysroot/nix/.rw-store. lb # [ 4.015625] fuse: init (API version 7.45) lb # [ 4.023176] virtiofs virtio6: discovered new tag: nix-store lb # [ 4.024078] virtiofs virtio6: virtio_fs_setup_dax: No cache capability client # [ 4.027534] fuse: init (API version 7.45) lb # [ 4.014251] systemd[1]: Starting rw-sysroot-nix-store.service... worker2 # [ 4.023943] systemd[1]: Mounting /sysroot/tmp/xchg... worker1 # [ 4.026479] systemd[1]: Mounted /sysroot/nix/.rw-store. lb # [ 4.041215] virtiofs virtio7: discovered new tag: shared client # [ 4.043995] virtiofs virtio6: discovered new tag: nix-store client # [ 4.044889] virtiofs virtio6: virtio_fs_setup_dax: No cache capability lb # [ 4.041970] virtiofs virtio7: virtio_fs_setup_dax: No cache capability client # [ 4.040375] systemd[1]: Mounted /sysroot/run. lb # [ 4.051580] virtiofs virtio8: discovered new tag: xchg worker1 # [ 4.051235] fuse: init (API version 7.45) lb # [ 4.052299] virtiofs virtio8: virtio_fs_setup_dax: No cache capability client # [ 4.059299] virtiofs virtio7: discovered new tag: shared client # [ 4.060072] virtiofs virtio7: virtio_fs_setup_dax: No cache capability worker1 # [ 4.056951] virtiofs virtio6: discovered new tag: nix-store lb # [ 4.049810] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker1 # [ 4.057979] virtiofs virtio6: virtio_fs_setup_dax: No cache capability client # [ 4.070062] virtiofs virtio8: discovered new tag: xchg client # [ 4.070791] virtiofs virtio8: virtio_fs_setup_dax: No cache capability lb # [ 4.056227] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 4.060097] systemd[1]: Mounted /sysroot/nix/.ro-store. client # [ 4.064149] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 4.072665] virtiofs virtio7: discovered new tag: shared lb # [ 4.062066] systemd[1]: Mounted /sysroot/tmp/shared. worker1 # [ 4.073406] virtiofs virtio7: virtio_fs_setup_dax: No cache capability lb # [ 4.065015] systemd[1]: Mounted /sysroot/tmp/xchg. worker1 # [ 4.081014] virtiofs virtio8: discovered new tag: xchg worker1 # [ 4.081719] virtiofs virtio8: virtio_fs_setup_dax: No cache capability lb # [ 4.071462] systemd[1]: Mounting /sysroot/nix/store... worker2 # [ 4.085131] fuse: init (API version 7.45) worker1 # [ 4.071777] systemd[1]: Starting rw-sysroot-nix-store.service... worker2 # [ 4.076376] systemd[1]: Mounted /sysroot/run. client # [ 4.078196] systemd[1]: Mounted /sysroot/tmp/xchg. client # [ 4.082245] systemd[1]: Mounted /sysroot/nix/.ro-store. worker2 # [ 4.095845] virtiofs virtio6: discovered new tag: nix-store worker1 # [ 4.082710] systemd[1]: Mounted /sysroot/run. worker2 # [ 4.096784] virtiofs virtio6: virtio_fs_setup_dax: No cache capability client # [ 4.088788] systemd[1]: Mounted /sysroot/tmp/shared. worker2 # [ 4.096056] systemd[1]: Mounted /sysroot/nix/.rw-store. client # [ 4.100723] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 4.102051] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.114783] virtiofs virtio7: discovered new tag: shared worker2 # [ 4.115620] virtiofs virtio7: virtio_fs_setup_dax: No cache capability worker1 # [ 4.105422] systemd[1]: Mounted /sysroot/nix/.ro-store. worker1 # [ 4.109863] systemd[1]: Mounted /sysroot/tmp/shared. worker2 # [ 4.127179] virtiofs virtio8: discovered new tag: xchg lb # [ 4.116252] systemd[1]: Mounted /sysroot/nix/store. worker1 # [ 4.115410] systemd[1]: Mounted /sysroot/tmp/xchg. worker2 # [ 4.131540] virtiofs virtio8: virtio_fs_setup_dax: No cache capability worker1 # [ 4.118369] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 4.120200] systemd[1]: Reached target Initrd File Systems. lb # [ 4.121048] systemd[1]: Starting Find NixOS closure... worker2 # [ 4.124187] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 4.121641] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 4.122819] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker2 # [ 4.133010] systemd[1]: Mounted /sysroot/tmp/shared. worker2 # [ 4.138021] systemd[1]: Mounted /sysroot/nix/.ro-store. worker2 # [ 4.140199] systemd[1]: Mounted /sysroot/tmp/xchg. worker2 # [ 4.161969] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 4.163383] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 4.161585] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. lb # [ 4.174638] systemd[1]: Finished Find NixOS closure. lb # [ 4.175438] systemd[1]: Reached target Initrd Default Target. lb # [ 4.178258] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... lb # [ 4.205707] systemd[1]: Stopped target Initrd Default Target. lb # [ 4.209109] systemd[1]: Stopped target Basic System. lb # [ 4.210185] systemd[1]: Stopped target Initrd Root Device. lb # [ 4.212215] systemd[1]: Stopped target Path Units. lb # [ 4.216108] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. lb # [ 4.218226] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. lb # [ 4.224633] systemd[1]: Stopped target Slice Units. lb # [ 4.225408] systemd[1]: Stopped target Socket Units. lb # [ 4.226922] systemd[1]: Stopped target System Initialization. lb # [ 4.228970] systemd[1]: Stopped target Swaps. lb # [ 4.231724] systemd[1]: Stopped target Timer Units. lb # [ 4.240575] systemd[1]: dbus.socket: Deactivated successfully. lb # [ 4.241650] systemd[1]: Closed D-Bus System Message Bus Socket. lb # [ 4.242552] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. lb # [ 4.243553] systemd[1]: Stopped Find NixOS closure. lb # [ 4.252231] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 4.254311] systemd[1]: systemd-sysctl.service: Deactivated successfully. lb # [ 4.257580] systemd[1]: Stopped Apply Kernel Variables. lb # [ 4.260837] systemd[1]: systemd-modules-load.service: Deactivated successfully. lb # [ 4.262994] systemd[1]: Stopped Load Kernel Modules. lb # [ 4.268262] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. lb # [ 4.269409] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. lb # [ 4.270471] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. lb # [ 4.271465] systemd[1]: Stopped Create System Files and Directories. lb # [ 4.274743] systemd[1]: Stopped target Local File Systems. lb # [ 4.275648] systemd[1]: Stopped target Preparation for Local File Systems. lb # [ 4.278846] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. lb # [ 4.279933] systemd[1]: Stopped Coldplug All udev Devices. lb # [ 4.283022] systemd[1]: Stopping Rule-based Manager for Device Events and Files... lb # [ 4.284177] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.285193] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.285916] systemd[1]: initrd-cleanup.service: Deactivated successfully. lb # [ 4.286832] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. lb # [ 4.287723] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 4.292816] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 4.296209] systemd[1]: systemd-udevd.service: Deactivated successfully. lb # [ 4.297196] systemd[1]: Stopped Rule-based Manager for Device Events and Files. lb # [ 4.299850] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. lb # [ 4.301414] systemd[1]: Closed udev Control Socket. lb # [ 4.302110] systemd[1]: Starting Cleanup udev Database... lb # [ 4.302849] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. lb # [ 4.303879] systemd[1]: Stopped Create Static Device Nodes in /dev. lb # [ 4.304859] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. lb # [ 4.305995] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. lb # [ 4.307134] systemd[1]: kmod-static-nodes.service: Deactivated successfully. lb # [ 4.312224] systemd[1]: Stopped Create List of Static Device Nodes. lb # [ 4.332433] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. lb # [ 4.334241] systemd[1]: Finished Cleanup udev Database. lb # [ 4.336544] systemd[1]: Reached target Switch Root. lb # [ 4.339701] systemd[1]: Starting NixOS Activation... lb # [ 4.434393] initrd-nixos-activation-start[170]: booting system configuration /nix/store/kgjzg78npi7z07am2szqxkr41f06jp5w-nixos-system-lb-test lb # [ 4.468203] initrd-nixos-activation-start[170]: running activation script... worker1 # [ 4.707475] (udev-worker)[96]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker1 # [ 4.712634] (udev-worker)[96]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory lb # [ 4.721508] initrd-nixos-activation-start[193]: setting up /etc... client # [ 4.735163] (udev-worker)[98]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker1 # [ 4.733769] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.738433] (udev-worker)[98]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory worker1 # [ 4.735972] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 4.739441] systemd[1]: Stopping Virtual Console Setup... worker1 # [ 4.740343] systemd[1]: Starting Virtual Console Setup... worker1 # [ 4.757947] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.759028] systemd[1]: Stopped Virtual Console Setup. client # [ 4.764142] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.760382] systemd[1]: Starting Virtual Console Setup... client # [ 4.766912] systemd[1]: Stopped Virtual Console Setup. client # [ 4.771630] systemd[1]: Stopping Virtual Console Setup... client # [ 4.772566] systemd[1]: Starting Virtual Console Setup... worker1 # [ 4.781187] systemd-vconsole-setup[153]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 4.784601] systemd[1]: Finished Virtual Console Setup. client # [ 4.796657] systemd-vconsole-setup[154]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 4.799829] systemd[1]: Finished Virtual Console Setup. lb # [ 4.841636] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. lb # [ 4.844173] systemd[1]: Finished NixOS Activation. worker2 # [ 4.846155] (udev-worker)[99]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory lb # [ 4.845376] systemd[1]: Starting Switch Root... worker2 # [ 4.849011] (udev-worker)[99]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory lb # [ 4.869376] systemd[1]: Switching root. worker2 # [ 4.872480] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 4.876134] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 4.877209] systemd[1]: Stopping Virtual Console Setup... worker2 # [ 4.880226] systemd[1]: Starting Virtual Console Setup... client # [ 4.885506] systemd[1]: Mounting /sysroot/nix/store... worker2 # [ 4.891011] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 4.892813] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 4.897889] systemd[1]: Starting Virtual Console Setup... worker1 # [ 4.913067] systemd[1]: Mounting /sysroot/nix/store... worker2 # [ 4.926217] systemd-vconsole-setup[153]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 4.929218] systemd[1]: Finished Virtual Console Setup. client # [ 4.939038] systemd[1]: Mounted /sysroot/nix/store. worker2 # [ 4.939525] systemd[1]: Mounting /sysroot/nix/store... client # [ 4.941368] systemd[1]: Reached target Initrd File Systems. client # [ 4.945187] systemd[1]: Starting Find NixOS closure... client # [ 4.956238] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker2 # [ 4.969378] systemd[1]: Mounted /sysroot/nix/store. worker2 # [ 4.970158] systemd[1]: Reached target Initrd File Systems. worker2 # [ 4.971320] systemd[1]: Starting Find NixOS closure... worker2 # [ 4.979613] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker1 # [ 4.980535] systemd[1]: Mounted /sysroot/nix/store. worker1 # [ 4.982605] systemd[1]: Reached target Initrd File Systems. client # [ 4.987909] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker1 # [ 4.986387] systemd[1]: Starting Find NixOS closure... client # [ 4.992294] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker1 # [ 4.993945] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker2 # [ 5.005407] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. client # [ 5.006562] systemd[1]: Finished Find NixOS closure. client # [ 5.009352] systemd[1]: Reached target Initrd Default Target. worker2 # [ 5.009426] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. client # [ 5.012245] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker2 # [ 5.019644] systemd[1]: Finished Find NixOS closure. worker2 # [ 5.021837] systemd[1]: Reached target Initrd Default Target. worker2 # [ 5.024178] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... client # [ 5.038680] systemd[1]: Stopped target Initrd Default Target. client # [ 5.040902] systemd[1]: Stopped target Basic System. lb # [ 5.052157] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker2 # [ 5.043521] systemd[1]: Stopped target Initrd Default Target. client # [ 5.044478] systemd[1]: Stopped target Initrd Root Device. worker1 # [ 5.038722] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. client # [ 5.045581] systemd[1]: Stopped target Path Units. worker2 # [ 5.045047] systemd[1]: Stopped target Basic System. client # [ 5.047985] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker1 # [ 5.042523] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker2 # [ 5.048425] systemd[1]: Stopped target Initrd Root Device. worker2 # [ 5.049264] systemd[1]: Stopped target Path Units. worker2 # [ 5.050072] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. client # [ 5.052194] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker2 # [ 5.051841] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. client # [ 5.053592] systemd[1]: Stopped target Slice Units. client # [ 5.055311] systemd[1]: Stopped target Socket Units. worker2 # [ 5.057242] systemd[1]: Stopped target Slice Units. client # [ 5.058474] systemd[1]: Stopped target System Initialization. client # [ 5.059434] systemd[1]: Stopped target Swaps. worker1 # [ 5.055342] systemd[1]: Finished Find NixOS closure. worker2 # [ 5.060552] systemd[1]: Stopped target Socket Units. worker2 # [ 5.062025] systemd[1]: Stopped target System Initialization. worker1 # [ 5.058175] systemd[1]: Reached target Initrd Default Target. client # [ 5.065350] systemd[1]: Stopped target Timer Units. worker1 # [ 5.060188] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker2 # [ 5.064929] systemd[1]: Stopped target Swaps. client # [ 5.066147] systemd[1]: dbus.socket: Deactivated successfully. client # [ 5.067016] systemd[1]: Closed D-Bus System Message Bus Socket. client # [ 5.067897] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker2 # [ 5.070482] systemd[1]: Stopped target Timer Units. worker2 # [ 5.071282] systemd[1]: dbus.socket: Deactivated successfully. worker2 # [ 5.072550] systemd[1]: Closed D-Bus System Message Bus Socket. client # [ 5.077525] systemd[1]: Stopped Find NixOS closure. worker2 # [ 5.076860] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker2 # [ 5.078020] systemd[1]: Stopped Find NixOS closure. client # [ 5.078942] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 5.079819] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker2 # [ 5.079318] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 5.082297] systemd[1]: Stopped Apply Kernel Variables. client # [ 5.083091] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker2 # [ 5.086741] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker2 # [ 5.087738] systemd[1]: Stopped Apply Kernel Variables. client # [ 5.090591] systemd[1]: Stopped Load Kernel Modules. worker2 # [ 5.090866] systemd[1]: systemd-modules-load.service: Deactivated successfully. client # [ 5.093853] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker1 # [ 5.089752] systemd[1]: Stopped target Initrd Default Target. worker2 # [ 5.094179] systemd[1]: Stopped Load Kernel Modules. worker1 # [ 5.091732] systemd[1]: Stopped target Basic System. client # [ 5.096617] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker2 # [ 5.096522] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker1 # [ 5.093317] systemd[1]: Stopped target Initrd Root Device. client # [ 5.099136] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker2 # [ 5.099026] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker1 # [ 5.096376] systemd[1]: Stopped target Path Units. worker2 # [ 5.101852] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker1 # [ 5.098085] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. client # [ 5.103369] systemd[1]: Stopped Create System Files and Directories. client # [ 5.104465] systemd[1]: Stopped target Local File Systems. worker2 # [ 5.104178] systemd[1]: Stopped Create System Files and Directories. client # [ 5.105325] systemd[1]: Stopped target Preparation for Local File Systems. worker2 # [ 5.105483] systemd[1]: Stopped target Local File Systems. worker1 # [ 5.101552] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. client # [ 5.106982] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. client # [ 5.108513] systemd[1]: Stopped Coldplug All udev Devices. worker2 # [ 5.108180] systemd[1]: Stopped target Preparation for Local File Systems. client # [ 5.109323] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker2 # [ 5.109190] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. client # [ 5.110342] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 5.111348] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 5.110750] systemd[1]: Stopped Coldplug All udev Devices. worker2 # [ 5.111631] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker1 # [ 5.108154] systemd[1]: Stopped target Slice Units. worker1 # [ 5.109122] systemd[1]: Stopped target Socket Units. worker2 # [ 5.112942] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 5.110039] systemd[1]: Stopped target System Initialization. worker2 # [ 5.113976] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 5.111081] systemd[1]: Stopped target Swaps. client # [ 5.115726] systemd[1]: initrd-cleanup.service: Deactivated successfully. worker2 # [ 5.114918] systemd[1]: initrd-cleanup.service: Deactivated successfully. worker1 # [ 5.112102] systemd[1]: Stopped target Timer Units. worker2 # [ 5.115843] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. client # [ 5.118031] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. client # [ 5.120399] systemd[1]: systemd-udevd.service: Deactivated successfully. worker1 # [ 5.116153] systemd[1]: dbus.socket: Deactivated successfully. worker2 # [ 5.122368] systemd[1]: systemd-udevd.service: Deactivated successfully. client # [ 5.124207] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker2 # [ 5.124165] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker1 # [ 5.120835] systemd[1]: Closed D-Bus System Message Bus Socket. client # [ 5.125237] systemd[1]: systemd-udevd.service: Consumed 1.524s CPU time over 3.040s wall clock time, 22.3M memory peak. worker1 # [ 5.122578] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. client # [ 5.128132] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. client # [ 5.129136] systemd[1]: Closed udev Control Socket. worker2 # [ 5.128349] systemd[1]: systemd-udevd.service: Consumed 1.548s CPU time over 2.961s wall clock time, 22.5M memory peak. worker1 # [ 5.126217] systemd[1]: Stopped Find NixOS closure. worker2 # [ 5.129942] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. client # [ 5.132262] systemd[1]: Starting Cleanup udev Database... worker2 # [ 5.132159] systemd[1]: Closed udev Control Socket. client # [ 5.133054] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker1 # [ 5.129343] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 5.136213] systemd[1]: Stopped Create Static Device Nodes in /dev. worker2 # [ 5.136108] systemd[1]: Starting Cleanup udev Database... client # [ 5.137107] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker2 # [ 5.136952] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker2 # [ 5.138036] systemd[1]: Stopped Create Static Device Nodes in /dev. worker2 # [ 5.138912] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. client # [ 5.140294] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. client # [ 5.141300] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker1 # [ 5.137746] systemd[1]: systemd-sysctl.service: Deactivated successfully. client # [ 5.144421] systemd[1]: Stopped Create List of Static Device Nodes. worker1 # [ 5.140320] systemd[1]: Stopped Apply Kernel Variables. client # [ 5.145340] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 5.144350] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. worker2 # [ 5.145383] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker2 # [ 5.146351] systemd[1]: Stopped Create List of Static Device Nodes. worker1 # [ 5.143194] systemd[1]: systemd-modules-load.service: Deactivated successfully. client # [ 5.148322] systemd[1]: Finished rw-sysroot-nix-store.service. worker1 # [ 5.144462] systemd[1]: Stopped Load Kernel Modules. worker2 # [ 5.148173] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker1 # [ 5.146215] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker2 # [ 5.152500] systemd[1]: Finished rw-sysroot-nix-store.service. worker1 # [ 5.149147] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker1 # [ 5.150256] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker1 # [ 5.151390] systemd[1]: Stopped Create System Files and Directories. worker1 # [ 5.156225] systemd[1]: Stopped target Local File Systems. worker1 # [ 5.158139] systemd[1]: Stopped target Preparation for Local File Systems. worker1 # [ 5.160231] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. client # [ 5.166521] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. client # [ 5.167635] systemd[1]: Finished Cleanup udev Database. worker1 # [ 5.163686] systemd[1]: Stopped Coldplug All udev Devices. worker1 # [ 5.165022] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker1 # [ 5.166039] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 5.167019] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 5.170819] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. client # [ 5.172315] systemd[1]: Reached target Switch Root. worker1 # [ 5.167746] systemd[1]: initrd-cleanup.service: Deactivated successfully. client # [ 5.173032] systemd[1]: Starting NixOS Activation... worker1 # [ 5.168862] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker1 # [ 5.169748] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 5.173781] systemd[1]: Finished Cleanup udev Database. worker1 # [ 5.170766] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 5.175024] systemd[1]: Reached target Switch Root. worker1 # [ 5.171572] systemd[1]: systemd-udevd.service: Deactivated successfully. worker2 # [ 5.179175] systemd[1]: Starting NixOS Activation... worker1 # [ 5.176278] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker1 # [ 5.180239] systemd[1]: systemd-udevd.service: Consumed 1.498s CPU time over 3.109s wall clock time, 25.2M memory peak. worker1 # [ 5.181918] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. worker1 # [ 5.184168] systemd[1]: Closed udev Control Socket. worker1 # [ 5.188162] systemd[1]: Starting Cleanup udev Database... worker1 # [ 5.188964] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker1 # [ 5.190003] systemd[1]: Stopped Create Static Device Nodes in /dev. worker1 # [ 5.190857] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker1 # [ 5.196358] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. worker1 # [ 5.197371] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker1 # [ 5.198320] systemd[1]: Stopped Create List of Static Device Nodes. worker1 # [ 5.221953] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker1 # [ 5.224792] systemd[1]: Finished Cleanup udev Database. worker1 # [ 5.228273] systemd[1]: Reached target Switch Root. worker1 # [ 5.229048] systemd[1]: Starting NixOS Activation... client # [ 5.248760] initrd-nixos-activation-start[178]: booting system configuration /nix/store/pka8rsnbb5rl489pq588cv7y1qycbh45-nixos-system-client-test worker2 # [ 5.254820] initrd-nixos-activation-start[178]: booting system configuration /nix/store/gnimypw801f4lzy6a5zb7cjxr1n5hqxf-nixos-system-worker2-test client # [ 5.279499] initrd-nixos-activation-start[178]: running activation script... worker2 # [ 5.286152] initrd-nixos-activation-start[178]: running activation script... worker1 # [ 5.305127] initrd-nixos-activation-start[176]: booting system configuration /nix/store/i9c4dcq9i981i967gfs22mhmyydrkc8s-nixos-system-worker1-test worker1 # [ 5.337077] initrd-nixos-activation-start[176]: running activation script... client # [ 5.529917] initrd-nixos-activation-start[201]: setting up /etc... worker2 # [ 5.528894] initrd-nixos-activation-start[201]: setting up /etc... worker1 # [ 5.559757] initrd-nixos-activation-start[199]: setting up /etc... client # [ 5.644125] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. client # [ 5.645254] systemd[1]: Finished NixOS Activation. worker2 # [ 5.644932] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. client # [ 5.647490] systemd[1]: Starting Switch Root... worker2 # [ 5.647636] systemd[1]: Finished NixOS Activation. worker2 # [ 5.648948] systemd[1]: Starting Switch Root... client # [ 5.670828] systemd[1]: Switching root. worker2 # [ 5.671176] systemd[1]: Switching root. worker1 # [ 5.674208] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. lb # [ 5.679072] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker1 # [ 5.676929] systemd[1]: Finished NixOS Activation. worker1 # [ 5.678086] systemd[1]: Starting Switch Root... lb # [ 5.692133] systemd[1]: Detected virtualization qemu. lb # [ 5.695046] systemd[1]: Detected architecture arm64. lb # [ 5.697930] systemd[1]: Detected first boot. lb # [ 5.703749] systemd[1]: Initializing machine ID from random generator. worker1 # [ 5.698808] systemd[1]: Switching root. client # [ 5.854680] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker2 # [ 5.857928] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker1 # [ 5.879106] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). lb # [ 6.029514] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 6.224057] systemd[1]: Applying preset policy. client # [ 6.384164] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 6.396154] systemd[1]: Detected virtualization qemu. worker2 # [ 6.387011] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 6.399135] systemd[1]: Detected architecture arm64. worker2 # [ 6.399784] systemd[1]: Detected virtualization qemu. client # [ 6.402745] systemd[1]: Detected first boot. worker2 # [ 6.402896] systemd[1]: Detected architecture arm64. client # [ 6.408206] systemd[1]: Initializing machine ID from random generator. worker2 # [ 6.406907] systemd[1]: Detected first boot. worker2 # [ 6.412758] systemd[1]: Initializing machine ID from random generator. worker1 # [ 6.402524] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker1 # [ 6.415016] systemd[1]: Detected virtualization qemu. worker1 # [ 6.417987] systemd[1]: Detected architecture arm64. worker1 # [ 6.421813] systemd[1]: Detected first boot. worker1 # [ 6.427300] systemd[1]: Initializing machine ID from random generator. lb # [ 6.461537] systemd[1]: Populated /etc with preset unit settings. lb # [ 6.698030] systemd[1]: initrd-switch-root.service: Deactivated successfully. lb # [ 6.699871] systemd[1]: Stopped initrd-switch-root.service. lb # [ 6.704051] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. lb # [ 6.708228] systemd[1]: Created slice Slice /system/getty. lb # [ 6.710370] systemd[1]: Created slice User and Session Slice. lb # [ 6.711641] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 6.714343] systemd[1]: Started Forward Password Requests to Wall Directory Watch. lb # [ 6.716532] systemd[1]: Expecting device /dev/hvc0... lb # [ 6.718527] systemd[1]: Expecting device /dev/ttyAMA0... lb # [ 6.720377] systemd[1]: Reached target Local Encrypted Volumes. lb # [ 6.722321] systemd[1]: Stopped target initrd-fs.target. lb # [ 6.724178] systemd[1]: Stopped target initrd-root-fs.target. lb # [ 6.726057] systemd[1]: Stopped target initrd-switch-root.target. client # [ 6.727560] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 6.728040] systemd[1]: Reached target Virtual Machines and Containers. lb # [ 6.730044] systemd[1]: Reached target Path Units. lb # [ 6.732039] systemd[1]: Reached target Remote File Systems. lb # [ 6.733900] systemd[1]: Reached target Slice Units. worker2 # [ 6.733823] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 6.735705] systemd[1]: Reached target Swaps. lb # [ 6.738873] systemd[1]: Listening on Query the User Interactively for a Password. lb # [ 6.741762] systemd[1]: Listening on Process Core Dump Socket. lb # [ 6.744059] systemd[1]: Listening on Credential Encryption/Decryption. lb # [ 6.746449] systemd[1]: Listening on Factory Reset Management. lb # [ 6.747703] systemd[1]: Listening on Hostname Service Socket. lb # [ 6.751590] systemd[1]: Starting Journal Log Access Socket... worker1 # [ 6.748635] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 6.753626] systemd[1]: Listening on Journal Audit Socket. lb # [ 6.757365] systemd[1]: Listening on Console Output Muting Service Socket. lb # [ 6.758950] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. lb # [ 6.761311] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os lb # [ 6.763012] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki lb # [ 6.769470] systemd[1]: Listening on Disk Repartitioning Service Socket. lb # [ 6.770896] systemd[1]: Listening on udev Control Socket. lb # [ 6.772443] systemd[1]: Listening on udev Varlink Socket. lb # [ 6.776354] systemd[1]: Mounting Huge Pages File System... lb # [ 6.783192] systemd[1]: Mounting POSIX Message Queue File System... lb # [ 6.793494] systemd[1]: Mounting Kernel Debug File System... lb # [ 6.802819] systemd[1]: Mounting Kernel Trace File System... lb # [ 6.815268] systemd[1]: Starting Create List of Static Device Nodes... lb # [ 6.816989] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 6.826949] systemd[1]: Mounting Kernel Configuration File System... lb # [ 6.830527] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm lb # [ 6.835772] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore lb # [ 6.843320] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 6.870075] systemd[1]: Mounting FUSE Control File System... lb # [ 6.874380] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 6.900047] systemd[1]: Starting Journal Service... lb # [ 6.914390] systemd[1]: Starting Load Kernel Modules... client # [ 6.932885] systemd[1]: Applying preset policy. worker2 # [ 6.932709] systemd[1]: Applying preset policy. lb # [ 6.934477] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... lb # [ 6.942369] systemd[1]: Starting Remount Root and Kernel File Systems... lb # [ 6.942720] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 6.956639] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 6.956200] systemd[1]: Applying preset policy. lb # [ 6.986478] systemd-journald[264]: Collecting audit messages is enabled. lb # [ 6.997481] systemd[1]: Listening on Journal Log Access Socket. lb # [ 6.997938] systemd[1]: Mounted Huge Pages File System. lb # [ 7.000760] systemd[1]: Mounted POSIX Message Queue File System. lb # [ 7.002742] systemd[1]: Mounted Kernel Debug File System. lb # [ 7.003967] systemd[1]: Mounted Kernel Trace File System. lb # [ 7.011522] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 7.012913] systemd[1]: Mounted Kernel Configuration File System. lb # [ 7.015819] systemd[1]: Mounted FUSE Control File System. lb # [ 7.003335] systemd[1]: Queued start job for default target Multi-User System. lb # [ 7.021754] systemd[1]: Started Journal Service. lb # [ 7.066355] EXT4-fs (vda): re-mounted 8ad90e9a-c614-4d36-9ae9-6371c810d3cf. worker2 # [ 7.190224] systemd[1]: Populated /etc with preset unit settings. client # [ 7.193028] systemd[1]: Populated /etc with preset unit settings. worker1 # [ 7.205047] systemd[1]: Populated /etc with preset unit settings. client # [ 7.414962] systemd[1]: initrd-switch-root.service: Deactivated successfully. client # [ 7.416186] systemd[1]: Stopped initrd-switch-root.service. client # [ 7.420418] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. client # [ 7.424723] systemd[1]: Created slice Slice /system/getty. client # [ 7.426668] systemd[1]: Created slice User and Session Slice. worker2 # [ 7.424864] systemd[1]: initrd-switch-root.service: Deactivated successfully. client # [ 7.427873] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker2 # [ 7.426473] systemd[1]: Stopped initrd-switch-root.service. client # [ 7.430590] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker1 # [ 7.428754] systemd[1]: initrd-switch-root.service: Deactivated successfully. client # [ 7.432690] systemd[1]: Expecting device /dev/hvc0... worker2 # [ 7.430671] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker1 # [ 7.430012] systemd[1]: Stopped initrd-switch-root.service. client # [ 7.434593] systemd[1]: Expecting device /dev/ttyAMA0... client # [ 7.436404] systemd[1]: Reached target Local Encrypted Volumes. worker2 # [ 7.435215] systemd[1]: Created slice Slice /system/getty. client # [ 7.438372] systemd[1]: Stopped target initrd-fs.target. worker2 # [ 7.437342] systemd[1]: Created slice User and Session Slice. worker1 # [ 7.434629] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. client # [ 7.440166] systemd[1]: Stopped target initrd-root-fs.target. worker1 # [ 7.438561] systemd[1]: Created slice Slice /system/getty. worker2 # [ 7.438577] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 7.442097] systemd[1]: Stopped target initrd-switch-root.target. worker1 # [ 7.440502] systemd[1]: Created slice User and Session Slice. worker2 # [ 7.441431] systemd[1]: Started Forward Password Requests to Wall Directory Watch. client # [ 7.443975] systemd[1]: Reached target Virtual Machines and Containers. worker2 # [ 7.443643] systemd[1]: Expecting device /dev/hvc0... client # [ 7.446058] systemd[1]: Reached target Path Units. worker1 # [ 7.441707] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker2 # [ 7.445537] systemd[1]: Expecting device /dev/ttyAMA0... client # [ 7.447973] systemd[1]: Reached target Remote File Systems. worker1 # [ 7.444394] systemd[1]: Started Forward Password Requests to Wall Directory Watch. client # [ 7.449878] systemd[1]: Reached target Slice Units. worker2 # [ 7.447430] systemd[1]: Reached target Local Encrypted Volumes. worker1 # [ 7.446770] systemd[1]: Expecting device /dev/hvc0... worker2 # [ 7.449506] systemd[1]: Stopped target initrd-fs.target. client # [ 7.451615] systemd[1]: Reached target Swaps. worker1 # [ 7.448624] systemd[1]: Expecting device /dev/ttyAMA0... worker2 # [ 7.450531] systemd[1]: Stopped target initrd-root-fs.target. client # [ 7.454652] systemd[1]: Listening on Query the User Interactively for a Password. worker1 # [ 7.450581] systemd[1]: Reached target Local Encrypted Volumes. worker2 # [ 7.452259] systemd[1]: Stopped target initrd-switch-root.target. worker2 # [ 7.454623] systemd[1]: Reached target Virtual Machines and Containers. worker1 # [ 7.452549] systemd[1]: Stopped target initrd-fs.target. client # [ 7.457483] systemd[1]: Listening on Process Core Dump Socket. worker1 # [ 7.454486] systemd[1]: Stopped target initrd-root-fs.target. worker2 # [ 7.455816] systemd[1]: Reached target Path Units. client # [ 7.459633] systemd[1]: Listening on Credential Encryption/Decryption. worker1 # [ 7.455552] systemd[1]: Stopped target initrd-switch-root.target. worker2 # [ 7.457272] systemd[1]: Reached target Remote File Systems. client # [ 7.461929] systemd[1]: Listening on Factory Reset Management. worker2 # [ 7.458808] systemd[1]: Reached target Slice Units. worker1 # [ 7.457170] systemd[1]: Reached target Virtual Machines and Containers. worker2 # [ 7.461160] systemd[1]: Reached target Swaps. worker1 # [ 7.459759] systemd[1]: Reached target Path Units. client # [ 7.463092] systemd[1]: Listening on Hostname Service Socket. worker1 # [ 7.460700] systemd[1]: Reached target Remote File Systems. worker2 # [ 7.463446] systemd[1]: Listening on Query the User Interactively for a Password. client # [ 7.466854] systemd[1]: Starting Journal Log Access Socket... worker1 # [ 7.462308] systemd[1]: Reached target Slice Units. worker1 # [ 7.464533] systemd[1]: Reached target Swaps. worker2 # [ 7.466457] systemd[1]: Listening on Process Core Dump Socket. client # [ 7.468808] systemd[1]: Listening on Journal Audit Socket. worker1 # [ 7.466799] systemd[1]: Listening on Query the User Interactively for a Password. worker2 # [ 7.468720] systemd[1]: Listening on Credential Encryption/Decryption. client # [ 7.472288] systemd[1]: Listening on Console Output Muting Service Socket. worker2 # [ 7.471037] systemd[1]: Listening on Factory Reset Management. worker1 # [ 7.469595] systemd[1]: Listening on Process Core Dump Socket. client # [ 7.473885] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker2 # [ 7.472298] systemd[1]: Listening on Hostname Service Socket. worker1 # [ 7.471846] systemd[1]: Listening on Credential Encryption/Decryption. client # [ 7.476012] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.474183] systemd[1]: Listening on Factory Reset Management. client # [ 7.478167] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 7.476677] systemd[1]: Starting Journal Log Access Socket... worker1 # [ 7.475363] systemd[1]: Listening on Hostname Service Socket. worker2 # [ 7.478249] systemd[1]: Listening on Journal Audit Socket. client # [ 7.482760] systemd[1]: Listening on Disk Repartitioning Service Socket. worker1 # [ 7.479213] systemd[1]: Starting Journal Log Access Socket... worker2 # [ 7.481001] systemd[1]: Listening on Console Output Muting Service Socket. client # [ 7.484052] systemd[1]: Listening on udev Control Socket. worker1 # [ 7.481154] systemd[1]: Listening on Journal Audit Socket. worker2 # [ 7.482534] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. client # [ 7.485528] systemd[1]: Listening on udev Varlink Socket. worker2 # [ 7.484090] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.483990] systemd[1]: Listening on Console Output Muting Service Socket. client # [ 7.489340] systemd[1]: Mounting Huge Pages File System... worker1 # [ 7.485512] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker2 # [ 7.487199] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker1 # [ 7.487129] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os client # [ 7.493577] systemd[1]: Mounting POSIX Message Queue File System... worker1 # [ 7.490055] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 7.492825] systemd[1]: Listening on Disk Repartitioning Service Socket. worker2 # [ 7.494164] systemd[1]: Listening on udev Control Socket. worker2 # [ 7.495661] systemd[1]: Listening on udev Varlink Socket. worker1 # [ 7.495656] systemd[1]: Listening on Disk Repartitioning Service Socket. worker1 # [ 7.496984] systemd[1]: Listening on udev Control Socket. worker2 # [ 7.498970] systemd[1]: Mounting Huge Pages File System... worker1 # [ 7.498525] systemd[1]: Listening on udev Varlink Socket. client # [ 7.505323] systemd[1]: Mounting Kernel Debug File System... worker1 # [ 7.502548] systemd[1]: Mounting Huge Pages File System... worker1 # [ 7.507740] systemd[1]: Mounting POSIX Message Queue File System... worker2 # [ 7.507985] systemd[1]: Mounting POSIX Message Queue File System... client # [ 7.516290] systemd[1]: Mounting Kernel Trace File System... worker2 # [ 7.516201] systemd[1]: Mounting Kernel Debug File System... worker1 # [ 7.516573] systemd[1]: Mounting Kernel Debug File System... worker2 # [ 7.525088] systemd[1]: Mounting Kernel Trace File System... client # [ 7.527548] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 7.527714] systemd[1]: Mounting Kernel Trace File System... client # [ 7.532712] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 7.537710] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 7.538950] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 7.540570] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 7.546184] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 7.553161] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 7.552005] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 7.553155] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm worker2 # [ 7.554456] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm client # [ 7.558120] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 7.554763] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore client # [ 7.559568] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm worker1 # [ 7.556435] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker2 # [ 7.561129] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker2 # [ 7.561487] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse client # [ 7.567663] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore client # [ 7.569611] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker2 # [ 7.576326] systemd[1]: Mounting FUSE Control File System... worker2 # [ 7.576691] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker1 # [ 7.575078] systemd[1]: Mounting FUSE Control File System... worker1 # [ 7.575443] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 client # [ 7.591451] systemd[1]: Mounting FUSE Control File System... client # [ 7.597235] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker1 # [ 7.598455] systemd[1]: Starting Journal Service... client # [ 7.611614] systemd[1]: Starting Journal Service... worker2 # [ 7.612186] systemd[1]: Starting Journal Service... client # [ 7.624023] systemd[1]: Starting Load Kernel Modules... worker1 # [ 7.627169] systemd[1]: Starting Load Kernel Modules... worker2 # [ 7.628128] systemd[1]: Starting Load Kernel Modules... client # [ 7.640127] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker2 # [ 7.641389] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker1 # [ 7.640407] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... lb # [ 7.005043] systemd[1]: systemd-journald.service: Deactivated successfully. lb # [ 7.643666] systemd-journald[264]: Received client request to flush runtime journal. worker2 # [ 7.656372] systemd[1]: Starting Remount Root and Kernel File Systems... worker1 # [ 7.654941] systemd[1]: Starting Remount Root and Kernel File Systems... client # [ 7.660454] systemd[1]: Starting Remount Root and Kernel File Systems... worker2 # [ 7.658145] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.656218] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 7.664831] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 7.663788] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 7.665172] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 7.667247] systemd[1]: Listening on Journal Log Access Socket. worker2 # [ 7.672228] systemd[1]: Mounted Huge Pages File System. worker1 # [ 7.669533] systemd[1]: Listening on Journal Log Access Socket. worker2 # [ 7.674561] systemd[1]: Mounted POSIX Message Queue File System. worker1 # [ 7.673492] systemd[1]: Mounted Huge Pages File System. worker2 # [ 7.677508] systemd[1]: Mounted Kernel Debug File System. worker1 # [ 7.676848] systemd[1]: Mounted POSIX Message Queue File System. worker1 # [ 7.679171] systemd[1]: Mounted Kernel Debug File System. worker2 # [ 7.680386] systemd[1]: Mounted Kernel Trace File System. worker2 # [ 7.680836] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 7.681435] systemd[1]: Mounted Kernel Trace File System. worker2 # [ 7.681214] systemd[1]: Mounted FUSE Control File System. client # [ 7.687821] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 7.684424] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 7.687273] systemd[1]: Mounted FUSE Control File System. client # [ 7.690797] systemd-journald[271]: Collecting audit messages is enabled. lb # [ 7.680902] systemd-modules-load[265]: Module 'atkbd' is built in client # [ 7.698887] systemd[1]: Listening on Journal Log Access Socket. lb # [ 7.682556] systemd-modules-load[265]: Module 'loop' is built in lb # [ 7.683651] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 7.702204] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 7.707410] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 7.709525] systemd[1]: Mounted Huge Pages File System. client # [ 7.710036] systemd[1]: Mounted POSIX Message Queue File System. worker1 # [ 7.707137] systemd[1]: Finished Create List of Static Device Nodes. client # [ 7.710424] systemd[1]: Mounted Kernel Debug File System. client # [ 7.710808] systemd[1]: Mounted Kernel Trace File System. lb # [ 7.699399] systemd[1]: Finished Load Kernel Modules. client # [ 7.703147] systemd[1]: Queued start job for default target Multi-User System. worker1 # [ 7.712220] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 7.718113] systemd[1]: Finished Create List of Static Device Nodes. client # [ 7.718663] systemd[1]: Started Journal Service. lb # [ 7.705301] systemd[1]: Starting Firewall... lb # [ 7.706115] systemd[1]: Starting Apply Kernel Variables... client # [ 7.710476] systemd[1]: systemd-journald.service: Deactivated successfully. lb # [ 7.707046] systemd[1]: Finished Remount Root and Kernel File Systems. lb # [ 7.716441] systemd[1]: Listening on Disk Image Download Service Socket. lb # [ 7.717545] systemd[1]: Starting Flush Journal to Persistent Storage... lb # [ 7.718583] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore client # [ 7.726053] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 7.727155] systemd[1]: Mounted Kernel Configuration File System. lb # [ 7.732449] systemd[1]: Starting Load/Save OS Random Seed... lb # [ 7.733432] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 7.735025] systemd-oomd[267]: No swap; memory pressure usage will be degraded client # [ 7.748601] systemd-modules-load[272]: Module 'atkbd' is built in client # [ 7.752872] systemd-modules-load[272]: Module 'loop' is built in lb # [ 7.749667] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. lb # [ 7.750724] systemd[1]: Finished Load/Save OS Random Seed. lb # [ 7.751629] systemd[1]: Reached target First Boot Complete. client # [ 7.755860] systemd-oomd[274]: No swap; memory pressure usage will be degraded worker2 # [ 7.768108] systemd[1]: Finished Load Kernel Modules. lb # [ 7.760888] systemd[1]: Finished Apply Kernel Variables. lb # [ 7.761706] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 7.762765] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 7.776879] systemd[1]: Starting Firewall... lb # [ 7.763656] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 7.769064] systemd[1]: Finished Load Kernel Modules. client # [ 7.772279] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker1 # [ 7.780018] systemd[1]: Finished Load Kernel Modules. worker2 # [ 7.783278] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 7.785904] systemd[1]: Starting Firewall... lb # [ 7.776179] systemd[1]: Reached target Preparation for Local File Systems. worker1 # [ 7.787873] EXT4-fs (vda): re-mounted 534bdfd8-5116-405d-9f63-cc51d12f4ec3. lb # [ 7.777200] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 7.778219] systemd[1]: Mounting /run/wrappers... lb # [ 7.778891] systemd[1]: Finished Flush Journal to Persistent Storage. client # [ 7.801256] EXT4-fs (vda): re-mounted 1fc68248-1ae2-42bc-8827-5f4f41b076d3. worker1 # [ 7.792685] systemd-journald[269]: Collecting audit messages is enabled. worker1 # [ 7.802243] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 7.803963] EXT4-fs (vda): re-mounted 849c7a3f-1ab2-4693-b52d-af9fe158a12b. client # [ 7.799556] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 7.809359] systemd[1]: Finished Remount Root and Kernel File Systems. worker1 # [ 7.807756] systemd[1]: Finished Remount Root and Kernel File Systems. client # [ 7.801818] systemd[1]: Finished Remount Root and Kernel File Systems. worker1 # [ 7.808330] systemd[1]: Listening on Disk Image Download Service Socket. worker2 # [ 7.811812] systemd[1]: Listening on Disk Image Download Service Socket. worker1 # [ 7.808616] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 7.812191] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore client # [ 7.808178] systemd[1]: Mounted FUSE Control File System. client # [ 7.809009] systemd[1]: Listening on Disk Image Download Service Socket. worker2 # [ 7.819423] systemd-journald[271]: Collecting audit messages is enabled. client # [ 7.813918] systemd[1]: Starting Flush Journal to Persistent Storage... worker1 # [ 7.822744] systemd[1]: Starting Load/Save OS Random Seed... client # [ 7.814872] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker1 # [ 7.824951] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 7.828563] systemd[1]: Starting Load/Save OS Random Seed... worker2 # [ 7.830693] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 7.825339] systemd[1]: Starting Load/Save OS Random Seed... client # [ 7.826306] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.854534] systemd[1]: Started Journal Service. worker1 # [ 7.843973] systemd[1]: Queued start job for default target Multi-User System. worker1 # [ 7.845243] systemd[1]: systemd-journald.service: Deactivated successfully. worker1 # [ 7.846419] systemd-modules-load[270]: Module 'atkbd' is built in worker1 # [ 7.847399] systemd-modules-load[270]: Module 'loop' is built in worker2 # [ 7.862177] systemd[1]: Started Journal Service. worker2 # [ 7.863853] systemd[1]: Queued start job for default target Multi-User System. worker2 # [ 7.865488] systemd[1]: systemd-journald.service: Deactivated successfully. worker2 # [ 7.876498] systemd-modules-load[272]: Module 'atkbd' is built in worker1 # [ 7.881375] systemd[1]: Starting Flush Journal to Persistent Storage... client # [ 7.902132] systemd-journald[271]: Received client request to flush runtime journal. worker2 # [ 7.888944] systemd-modules-load[272]: Module 'loop' is built in worker2 # [ 7.896974] systemd[1]: Starting Flush Journal to Persistent Storage... worker2 # [ 7.928387] systemd-oomd[273]: No swap; memory pressure usage will be degraded worker1 # [ 7.929098] systemd[1]: Finished Load/Save OS Random Seed. worker1 # [ 7.934210] systemd[1]: Reached target First Boot Complete. client # [ 7.944505] systemd[1]: Finished Load/Save OS Random Seed. client # [ 7.945422] systemd[1]: Reached target First Boot Complete. lb # [ 7.942142] systemd[1]: Mounted /run/wrappers. lb # [ 7.942970] systemd[1]: Reached target Local File Systems. client # [ 7.948867] systemd[1]: Finished Flush Journal to Persistent Storage. worker1 # [ 7.947900] systemd-oomd[272]: No swap; memory pressure usage will be degraded worker2 # [ 7.953319] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. lb # [ 7.951453] systemd[1]: Listening on Boot Loader Control Service Socket. lb # [ 7.957727] systemd[1]: Starting register-nix-paths.service... worker2 # [ 7.960601] systemd[1]: Finished Load/Save OS Random Seed. lb # [ 7.960255] systemd[1]: Starting Create SUID/SGID Wrappers... lb # [ 7.961116] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker2 # [ 7.964307] systemd[1]: Reached target First Boot Complete. worker2 # [ 7.968588] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 7.966993] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 7.978200] systemd[1]: Starting Save Transient machine-id to Disk... worker1 # [ 7.989602] systemd-journald[269]: Received client request to flush runtime journal. worker2 # [ 7.983253] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 7.982505] systemd-udevd[289]: Using default interface naming scheme 'v261'. worker2 # [ 7.985617] systemd[1]: Starting Create Static Device Nodes in /dev... lb # [ 7.993273] systemd[1]: Starting Create System Files and Directories... worker2 # [ 8.024190] systemd-journald[271]: Received client request to flush runtime journal. worker1 # [ 8.022756] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 8.023777] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. client # [ 8.032790] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 8.036480] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 8.037365] systemd[1]: Finished Flush Journal to Persistent Storage. worker2 # [ 8.085992] systemd[1]: Finished Flush Journal to Persistent Storage. lb # [ 8.094469] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. lb # [ 8.100208] systemd[1]: Finished Save Transient machine-id to Disk. client # [ 8.160788] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 8.163685] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 8.168800] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 8.168382] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 8.172692] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker1 # [ 8.182721] systemd[1]: Finished Create Static Device Nodes in /dev. worker1 # [ 8.189586] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 8.192403] systemd[1]: Finished Create System Files and Directories. worker1 # [ 8.196195] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 8.204299] systemd[1]: Starting Rebuild Journal Catalog... lb # [ 8.211815] systemd[1]: Starting Record System Boot/Shutdown in UTMP... lb # [ 8.246071] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 8.281409] systemd-udevd[311]: Using default interface naming scheme 'v261'. worker1 # [ 8.294283] systemd-udevd[307]: Using default interface naming scheme 'v261'. lb # [ 8.350281] systemd[1]: Finished Record System Boot/Shutdown in UTMP. lb # [ 8.367647] systemd[1]: Finished Coldplug All udev Devices. lb # [ 8.385300] systemd[1]: Finished Rebuild Journal Catalog. lb # [ 8.392728] systemd[1]: Starting Update is Completed... worker1 # [ 8.402595] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 8.415520] systemd[1]: Mounting /run/wrappers... worker1 # [ 8.421394] systemd[1]: Mounting /run/wrappers... worker2 # [ 8.461999] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 8.462392] systemd[1]: Finished Update is Completed. worker2 # [ 8.482964] systemd[1]: Mounted /run/wrappers. worker2 # [ 8.491224] systemd[1]: Reached target Local File Systems. worker2 # [ 8.492441] systemd[1]: Listening on Boot Loader Control Service Socket. worker2 # [ 8.499219] systemd[1]: Starting register-nix-paths.service... client # [ 8.506338] systemd[1]: Finished Coldplug All udev Devices. worker2 # [ 8.508391] systemd[1]: Starting Create SUID/SGID Wrappers... worker2 # [ 8.509213] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker2 # [ 8.510441] systemd[1]: Starting Save Transient machine-id to Disk... client # [ 8.514107] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 8.516364] systemd[1]: Reached target Preparation for Local File Systems. client # [ 8.519575] systemd[1]: Mounting /run/wrappers... client # [ 8.524534] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker2 # [ 8.533314] systemd[1]: Starting Create System Files and Directories... worker1 # [ 8.545514] systemd[1]: Mounted /run/wrappers. worker1 # [ 8.546456] systemd[1]: Reached target Local File Systems. worker1 # [ 8.550762] systemd[1]: Listening on Boot Loader Control Service Socket. worker1 # [ 8.559625] systemd[1]: Starting register-nix-paths.service... client # [ 8.566172] systemd[1]: Mounted /run/wrappers. client # [ 8.567200] systemd[1]: Reached target Local File Systems. client # [ 8.573266] systemd[1]: Listening on Boot Loader Control Service Socket. worker1 # [ 8.570793] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 8.576235] systemd[1]: Starting register-nix-paths.service... client # [ 8.581837] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 8.582853] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker1 # [ 8.580318] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker1 # [ 8.585037] systemd[1]: Starting Save Transient machine-id to Disk... client # [ 8.590532] systemd[1]: Starting Save Transient machine-id to Disk... worker1 # [ 8.589682] systemd[1]: Starting Create System Files and Directories... client # [ 8.609920] systemd-udevd[302]: Using default interface naming scheme 'v261'. worker2 # [ 8.617449] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. client # [ 8.619592] systemd[1]: Starting Create System Files and Directories... worker2 # [ 8.623080] systemd[1]: Finished Save Transient machine-id to Disk. client # [ 8.656434] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. client # [ 8.663259] systemd[1]: Finished Save Transient machine-id to Disk. worker1 # [ 8.677740] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker1 # [ 8.682856] systemd[1]: Finished Save Transient machine-id to Disk. client # [ 8.735031] systemd[1]: Finished Create System Files and Directories. worker2 # [ 8.737088] systemd[1]: Finished Create System Files and Directories. client # [ 8.743276] systemd[1]: Starting Rebuild Journal Catalog... worker2 # [ 8.742489] systemd[1]: Starting Rebuild Journal Catalog... client # [ 8.746859] systemd[1]: Starting Record System Boot/Shutdown in UTMP... worker2 # [ 8.753489] systemd[1]: Starting Record System Boot/Shutdown in UTMP... worker1 # [ 8.770666] systemd[1]: Finished Create System Files and Directories. client # [ 8.771175] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 8.778502] systemd[1]: Starting Rebuild Journal Catalog... worker1 # [ 8.789299] systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [ 8.840389] systemd[1]: Finished Record System Boot/Shutdown in UTMP. worker2 # [ 8.861921] systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 8.883869] systemd[1]: Finished Rebuild Journal Catalog. worker1 # [ 8.884784] systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 8.893110] systemd[1]: Starting Update is Completed... worker2 # [ 8.923213] systemd[1]: Finished Rebuild Journal Catalog. worker2 # [ 8.926703] systemd[1]: Starting Update is Completed... client # [ 8.961211] systemd[1]: Finished Update is Completed. worker1 # [ 8.964164] systemd[1]: Finished Rebuild Journal Catalog. worker1 # [ 8.972263] systemd[1]: Starting Update is Completed... worker2 # [ 9.012624] systemd[1]: Finished Update is Completed. lb # [ 9.025640] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. lb # [ 9.033553] systemd[1]: Finished Create SUID/SGID Wrappers. worker1 # [ 9.063441] systemd[1]: Finished Update is Completed. client # [ 9.151989] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 9.193132] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 9.194598] systemd[1]: Finished register-nix-paths.service. lb # [ 9.196202] systemd[1]: Reached target System Initialization. lb # [ 9.198457] systemd[1]: Started Discard unused filesystem blocks once a week. lb # [ 9.200962] systemd[1]: Started niks3 garbage collection timer. lb # [ 9.203070] systemd[1]: Started Daily Cleanup of Temporary Directories. lb # [ 9.207211] systemd[1]: Reached target Timer Units. lb # [ 9.208005] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 9.217470] systemd[1]: Listening on niks3 server socket. lb # [ 9.220755] systemd[1]: Listening on Nix Daemon Socket. lb # [ 9.221611] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. lb # [ 9.222747] systemd[1]: Reached target Socket Units. lb # [ 9.223438] systemd[1]: Reached target Basic System. lb # [ 9.231033] systemd[1]: Starting Import lastlog data into lastlog2 database... lb # [ 9.235181] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 9.250106] systemd[1]: Starting Post-Boot Actions... lb # [ 9.254599] systemd[1]: Started Reset console on configuration changes. lb # [ 9.295649] systemd[1]: Starting resolvconf update... lb # [ 9.324254] systemd[1]: Started rustfs.service. lb # [ 9.356122] systemd[1]: Starting rustfs-bucket.service... client # [ 9.362214] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. client # [ 9.365260] systemd[1]: Finished Create SUID/SGID Wrappers. lb # [ 9.376834] nsncd[445]: Sep 23 04:11:23.990 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 9.392283] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. client # [ 9.394539] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker2 # [ 9.394369] systemd[1]: Finished Create SUID/SGID Wrappers. lb # [ 9.397550] systemd[1]: Started Name Service Cache Daemon (nsncd). lb # [ 9.439583] systemd[1]: Finished Post-Boot Actions. lb # [ 9.455951] systemd[1]: Reached target Host and Network Name Lookups. client # [ 9.464285] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. lb # [ 9.464233] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 9.471188] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker1 # [ 9.477368] systemd[1]: Finished Create SUID/SGID Wrappers. lb # [ 9.485088] systemd[1]: Starting D-Bus System Message Bus... lb # [ 9.506441] systemd[1]: Starting User Login Management... lb # [ 9.543969] systemd[1]: Finished Import lastlog data into lastlog2 database. client # [ 9.631894] (udev-worker)[356]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 9.640308] (udev-worker)[356]: Network interface NamePolicy= disabled on kernel command line. client # [ 9.646202] (udev-worker)[350]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 9.683327] systemd[1]: Finished Firewall. worker1 # [ 9.732431] systemd[1]: Finished Firewall. worker1 # [ 9.750509] systemd[1]: Finished Coldplug All udev Devices. lb # [ 9.776691] dbus-broker-launch[477]: Looking up NSS user entry for 'systemd-timesync'... lb # [ 9.777916] dbus-broker-launch[477]: NSS returned no entry for 'systemd-timesync' lb # [ 9.779014] dbus-broker-launch[477]: Invalid user-name in /nix/store/4yiprls1sll47jid3xzq0m9nmi1qz2ww-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker1 # [ 9.824304] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 9.828981] systemd[1]: Stopped target Host and Network Name Lookups. lb # [ 9.831975] systemd[1]: Stopping Host and Network Name Lookups... lb # [ 9.837930] systemd[1]: Stopped target User and Group Name Lookups. lb # [ 9.838877] systemd[1]: Stopping User and Group Name Lookups... lb # [ 9.839712] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker2 # [ 9.849140] systemd[1]: Finished Coldplug All udev Devices. lb # [ 9.852197] systemd-logind[480]: New seat seat0. lb # [ 9.852947] systemd[1]: Started User Login Management. lb # [ 9.853705] systemd[1]: nscd.service: Deactivated successfully. lb # [ 9.854527] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker1 # [ 9.866342] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 9.874167] systemd[1]: Starting linger-users.service... client # [ 9.880420] systemd[1]: Condition check resulted in Virtio network device being skipped. client # [ 9.884920] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore client # [ 9.889320] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 9.890203] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. client # [ 9.891427] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 9.895765] systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 9.903763] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore client # [ 9.906129] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 9.907395] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 9.904486] systemd[1]: Started D-Bus System Message Bus. worker2 # [ 9.915455] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 9.934942] dbus-broker-launch[477]: Ready client # [ 9.941380] systemd[1]: Finished register-nix-paths.service. worker2 # [ 9.943842] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse client # [ 9.965420] mousedev: PS/2 mouse device common for all mice lb # [ 10.023355] systemd[1]: linger-users.service: Deactivated successfully. lb # [ 10.030370] systemd[1]: Finished linger-users.service. worker1 # [ 10.034273] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. lb # [ 10.042568] nsncd[549]: Sep 23 04:11:24.655 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" lb # [ 10.053772] systemd[1]: Started Name Service Cache Daemon (nsncd). lb # [ 10.054692] systemd[1]: Reached target Host and Network Name Lookups. lb # [ 10.055571] systemd[1]: Reached target User and Group Name Lookups. lb # [ 10.070693] systemd[1]: Finished resolvconf update. lb # [ 10.081061] systemd[1]: Starting DHCP Client... worker1 # [ 10.084957] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 10.103810] systemd[1]: Finished register-nix-paths.service. worker1 # [ 10.100530] systemd[1]: Finished register-nix-paths.service. worker2 # [ 10.105987] systemd[1]: Reached target System Initialization. worker1 # [ 10.103278] systemd[1]: Reached target System Initialization. worker2 # [ 10.107929] systemd[1]: Started Discard unused filesystem blocks once a week. worker1 # [ 10.106769] systemd[1]: Started Discard unused filesystem blocks once a week. worker1 # [ 10.109510] systemd[1]: Started Daily Cleanup of Temporary Directories. worker2 # [ 10.114385] systemd[1]: Started Daily Cleanup of Temporary Directories. worker1 # [ 10.111582] systemd[1]: Reached target Timer Units. worker1 # [ 10.116559] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 10.120517] systemd[1]: Reached target Timer Units. worker2 # [ 10.121324] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 10.122209] systemd[1]: Listening on Nix Daemon Socket. worker2 # [ 10.122948] systemd[1]: Listening on Nix worker-protocol over gRPC. worker2 # [ 10.123832] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker1 # [ 10.119646] systemd[1]: Listening on Nix Daemon Socket. worker1 # [ 10.130640] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 10.139026] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker2 # [ 10.146030] systemd[1]: Reached target Socket Units. worker1 # [ 10.147971] systemd[1]: Reached target Socket Units. worker1 # [ 10.152130] systemd[1]: Reached target Basic System. worker2 # [ 10.159251] systemd[1]: Reached target Basic System. worker1 # [ 10.157210] systemd[1]: Started backdoor.service. worker2 # [ 10.164195] systemd[1]: Starting Import lastlog data into lastlog2 database... worker2 # [ 10.167618] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 10.167623] systemd[1]: Starting Import lastlog data into lastlog2 database... worker2 # [ 10.177253] systemd[1]: Starting Post-Boot Actions... worker1 # [ 10.175267] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 10.183834] systemd[1]: Started Reset console on configuration changes. worker1 # [ 10.188227] systemd[1]: Starting Post-Boot Actions... worker2 # [ 10.195924] systemd[1]: Starting resolvconf update... worker1 # [ 10.201556] systemd[1]: Started Reset console on configuration changes. worker2 # [ 10.205842] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker1 # [ 10.208810] systemd[1]: Starting resolvconf update... worker2 # [ 10.216415] systemd[1]: Starting D-Bus System Message Bus... worker1 # [ 10.215631] systemd[1]: Starting D-Bus System Message Bus... worker2 # [ 10.258105] systemd[1]: Finished Post-Boot Actions. worker1 # [ 10.264507] systemd[1]: Finished Post-Boot Actions. worker2 # [ 10.288956] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 10.292894] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 10.300214] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 10.304484] nsncd[490]: Sep 23 04:11:24.870 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 10.307958] nsncd[489]: Sep 23 04:11:24.902 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # connecting to host... worker2 # [ 10.320107] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 10.326227] systemd[1]: Starting User Login Management... worker1 # [ 10.326211] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 10.331171] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 10.337592] systemd[1]: Started backdoor.service. worker1 # [ 10.334151] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 10.340284] systemd[1]: Starting User Login Management... lb # [ 10.347279] dhcpcd[603]: dhcpcd-10.3.2 starting worker2 # [ 10.355532] systemd[1]: Finished Import lastlog data into lastlog2 database. lb # [ 10.369176] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 lb # [ 10.357646] systemd[1]: Finished Firewall. lb # [ 10.359755] systemd[1]: Reached target Preparation for Network. lb # [ 10.365188] systemd[1]: Starting Extra networking commands.... worker1 # [ 10.364332] systemd[1]: Finished Import lastlog data into lastlog2 database. lb # [ 10.368322] dhcpcd[617]: dev: loaded udev worker2 # [ 10.371132] dbus-broker-launch[494]: Looking up NSS user entry for 'systemd-timesync'... client # [ 10.376113] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. lb # [ 10.376108] dhcpcd[617]: no valid interfaces found client # [ 10.379590] systemd[1]: Finished Create SUID/SGID Wrappers. lb # [ 10.377631] dhcpcd[617]: no valid interfaces found worker2 # [ 10.382492] dbus-broker-launch[494]: NSS returned no entry for 'systemd-timesync' client # [ 10.383845] systemd[1]: Reached target System Initialization. client # [ 10.385560] systemd[1]: Started Discard unused filesystem blocks once a week. client # [ 10.386552] systemd[1]: Started Daily Cleanup of Temporary Directories. client # [ 10.387452] systemd[1]: Reached target Timer Units. worker2 # [ 10.388509] dbus-broker-launch[494]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker1 # [ 10.386483] dbus-broker-launch[493]: Looking up NSS user entry for 'systemd-timesync'... client # [ 10.393397] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 10.396501] systemd[1]: Listening on Nix Daemon Socket. client # [ 10.399782] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker1 # [ 10.398904] dbus-broker-launch[493]: NSS returned no entry for 'systemd-timesync' client # [ 10.405393] systemd[1]: Reached target Socket Units. client # [ 10.406142] systemd[1]: Reached target Basic System. worker1 # [ 10.399928] dbus-broker-launch[493]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" client # [ 10.406880] systemd[1]: Started backdoor.service. client # [ 10.407567] systemd[1]: Starting Import lastlog data into lastlog2 database... worker2 # [ 10.407362] systemd[1]: Started D-Bus System Message Bus. client # [ 10.414582] systemd[1]: Starting nftables firewall... worker1 # [ 10.418905] systemd[1]: Started D-Bus System Message Bus. client # [ 10.432153] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # connecting to host... client # [ 10.454376] systemd[1]: Starting Post-Boot Actions... client # [ 10.463646] systemd[1]: Started Reset console on configuration changes. worker2 # [ 10.463903] dbus-broker-launch[494]: Ready worker1 # [ 10.460871] dbus-broker-launch[493]: Ready client # [ 10.480336] systemd[1]: Starting resolvconf update... worker1 # [ 10.505670] systemd-logind[503]: New seat seat0. worker1 # [ 10.513972] systemd[1]: Started User Login Management. worker1 # [ 10.514749] systemd[1]: Starting linger-users.service... worker2 # [ 10.525014] systemd-logind[502]: New seat seat0. worker2 # [ 10.532431] systemd[1]: Started User Login Management. worker2 # [ 10.533210] systemd[1]: Starting linger-users.service... worker1 # [ 10.537136] systemd[1]: Stopped target Host and Network Name Lookups. worker1 # [ 10.538082] systemd[1]: Stopping Host and Network Name Lookups... client # [ 10.543123] systemd[1]: Starting D-Bus System Message Bus... client # connecting to host... worker2 # [ 10.554724] systemd[1]: Stopped target Host and Network Name Lookups. worker2 # [ 10.555665] systemd[1]: Stopping Host and Network Name Lookups... worker1 # [ 10.554452] systemd[1]: Stopped target User and Group Name Lookups. worker1 # [ 10.561312] systemd[1]: Stopping User and Group Name Lookups... worker1 # [ 10.562171] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker1 # [ 10.563019] systemd[1]: nscd.service: Deactivated successfully. worker1 # [ 10.563799] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker2 # [ 10.568601] systemd[1]: Stopped target User and Group Name Lookups. worker2 # [ 10.569573] systemd[1]: Stopping User and Group Name Lookups... worker2 # [ 10.570486] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker2 # [ 10.571355] systemd[1]: nscd.service: Deactivated successfully. worker1 # [ 10.577101] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 10.578312] (udev-worker)[333]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 10.585554] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker2 # [ 10.586639] systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 10.587671] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 10.590900] nsncd[454]: Sep 23 04:11:25.194 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 10.591457] (udev-worker)[333]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 10.598270] (udev-worker)[418]: Network interface NamePolicy= disabled on kernel command line. client: Guest shell says: b'Spawning backdoor root shell...\n' worker1 # [ 10.609632] systemd[1]: linger-users.service: Deactivated successfully. worker1 # [ 10.610534] systemd[1]: Finished linger-users.service. client # [ 10.622286] systemd[1]: Finished Post-Boot Actions. worker2 # [ 10.620598] systemd[1]: linger-users.service: Deactivated successfully. worker1 # [ 10.644519] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 10.636383] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 10.621485] systemd[1]: Finished linger-users.service. worker1 # [ 10.645361] systemd[1]: Reached target Host and Network Name Lookups. client # [ 10.637286] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 10.651175] (udev-worker)[340]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client: connected to guest root shell worker1 # [ 10.646189] systemd[1]: Reached target User and Group Name Lookups. client # [ 10.661516] systemd[1]: Starting User Login Management... worker2 # [ 10.663322] systemd[1]: Started Name Service Cache Daemon (nsncd). client: (connecting took 11.10 seconds) worker2 # [ 10.667670] systemd[1]: Reached target Host and Network Name Lookups. client: (finished: waiting for the VM to finish booting, in 11.10 seconds) worker2 # [ 10.670297] nsncd[563]: Sep 23 04:11:25.243 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 10.655354] nsncd[561]: Sep 23 04:11:25.251 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 10.676563] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 10.677390] (udev-worker)[340]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 10.678728] (udev-worker)[339]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 10.684087] systemd[1]: Finished resolvconf update. worker2 # [ 10.690520] systemd[1]: Finished resolvconf update. worker2 # [ 10.691258] systemd[1]: Reached target Preparation for Network. worker1 # [ 10.687833] systemd[1]: Reached target Preparation for Network. worker1 # [ 10.692772] systemd[1]: Starting DHCP Client... worker2 # [ 10.703434] systemd[1]: Starting DHCP Client... worker1 # [ 10.700309] systemd[1]: Starting Extra networking commands.... worker2 # [ 10.705111] systemd[1]: Starting Extra networking commands.... client # [ 10.712784] systemd[1]: Finished Import lastlog data into lastlog2 database. lb # [ 10.732431] systemd[1]: Finished Extra networking commands.. lb # [ 10.736238] systemd[1]: Reached target Network. client # [ 10.746799] dbus-broker-launch[462]: Looking up NSS user entry for 'systemd-timesync'... lb # [ 10.746963] systemd[1]: Starting PostgreSQL Server... lb # [ 10.749493] systemd[1]: Starting Permit User Sessions... client # [ 10.761490] dbus-broker-launch[462]: NSS returned no entry for 'systemd-timesync' client # [ 10.762552] dbus-broker-launch[462]: Invalid user-name in /nix/store/6wvgzhp3xj1v1xybf1ixpbqi5gif5fbv-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" lb # [ 10.844483] systemd[1]: Finished Permit User Sessions. client # [ 10.858622] systemd[1]: Started D-Bus System Message Bus. lb # [ 10.860601] systemd[1]: Started Getty on tty1. lb # [ 10.861807] systemd[1]: Reached target Login Prompts. client # [ 10.895559] dbus-broker-launch[462]: Ready worker1 # [ 10.897097] dhcpcd[597]: dhcpcd-10.3.2 starting worker1 # [ 10.908210] dhcpcd[636]: dev: loaded udev client # [ 10.902350] systemd-logind[484]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) client # [ 10.925473] systemd-logind[484]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 10.926537] systemd-logind[484]: New seat seat0. worker2 # [ 10.930894] dhcpcd[598]: dhcpcd-10.3.2 starting client # [ 10.935140] systemd[1]: Started User Login Management. client # [ 10.938969] systemd[1]: Starting linger-users.service... worker2 # [ 10.940524] dhcpcd[641]: dev: loaded udev worker1 # [ 10.952539] 8021q: 802.1Q VLAN Support v1.8 client # [ 10.971177] systemd[1]: Stopped target Host and Network Name Lookups. worker2 # [ 10.986567] 8021q: 802.1Q VLAN Support v1.8 client # [ 10.977825] systemd[1]: Stopping Host and Network Name Lookups... client # [ 10.978723] systemd[1]: Stopped target User and Group Name Lookups. client # [ 10.979553] systemd[1]: Stopping User and Group Name Lookups... client # [ 10.988932] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... client # [ 10.989830] systemd[1]: nscd.service: Deactivated successfully. worker1 # [ 10.985331] systemd[1]: Condition check resulted in Virtio network device being skipped. client # [ 10.990635] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker1 # [ 10.996472] systemd[1]: Starting Address configuration of eth1... lb # [ 11.022322] postgresql-pre-start[677]: The files belonging to this database system will be owned by user "postgres". lb # [ 11.023786] postgresql-pre-start[677]: This user must also own the server process. worker2 # [ 11.027575] systemd[1]: Finished Extra networking commands.. worker2 # [ 11.032382] systemd[1]: Reached target Network. client # [ 11.032548] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 11.033090] systemd[1]: Starting Permit User Sessions... worker1 # [ 11.032177] systemd[1]: Finished Extra networking commands.. worker1 # [ 11.033068] systemd[1]: Reached target Network. lb # [ 11.041944] postgresql-pre-start[677]: The database cluster will be initialized with locale "en_US.UTF-8". worker1 # [ 11.054737] cfg80211: Loading compiled-in X.509 certificates for regulatory database lb # [ 11.043274] postgresql-pre-start[677]: The default database encoding has accordingly been set to "UTF8". worker1 # [ 11.036438] systemd-logind[503]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 11.050905] systemd[1]: linger-users.service: Deactivated successfully. lb # [ 11.048314] postgresql-pre-start[677]: The default text search configuration will be set to "english". lb # [ 11.049558] postgresql-pre-start[677]: Data page checksums are enabled. client # [ 11.053810] systemd[1]: Finished linger-users.service. worker1 # [ 11.049264] systemd[1]: Starting Permit User Sessions... lb # [ 11.050476] postgresql-pre-start[677]: fixing permissions on existing directory /var/lib/postgresql/18 ... ok lb # [ 11.051714] postgresql-pre-start[677]: creating subdirectories ... ok lb # [ 11.057675] postgresql-pre-start[677]: selecting dynamic shared memory implementation ... posix worker2 # [ 11.083751] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker2 # [ 11.083921] systemd[1]: Condition check resulted in Virtio network device being skipped. worker1 # [ 11.094662] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker1 # [ 11.095159] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker2 # [ 11.090296] systemd[1]: Starting Address configuration of eth1... worker1 # [ 11.099282] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker1 # [ 11.099605] cfg80211: failed to load regulatory.db lb # [ 11.097093] systemd[1]: Starting Virtual Console Setup... client # [ 11.103622] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 11.109261] systemd[1]: Reached target Host and Network Name Lookups. client # [ 11.110445] nsncd[545]: Sep 23 04:11:25.713 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 11.115716] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 11.132265] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker2 # [ 11.132761] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' lb # [ 11.120530] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 11.135288] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker2 # [ 11.135621] cfg80211: failed to load regulatory.db lb # [ 11.123410] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 11.128197] systemd[1]: Finished Permit User Sessions. client # [ 11.136125] systemd[1]: Finished resolvconf update. worker2 # [ 11.135661] systemd[1]: Finished Permit User Sessions. worker1 # [ 11.133252] systemd[1]: Started Getty on tty1. worker1 # [ 11.133979] systemd[1]: Reached target Login Prompts. lb # [ 11.136825] systemd[1]: Starting Virtual Console Setup... worker2 # [ 11.141346] systemd[1]: Started Getty on tty1. worker2 # [ 11.142017] systemd[1]: Reached target Login Prompts. client # [ 11.144695] systemd[1]: Starting DHCP Client... worker1 # [ 11.168660] 8021q: adding VLAN 0 to HW filter on device eth1 worker1 # [ 11.175954] network-addresses-eth1-start[649]: adding address 192.168.1.3/24... done worker1 # [ 11.192260] network-addresses-eth1-start[649]: adding address 2001:db8:1::3/64... done lb # [ 11.193297] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 11.210412] 8021q: adding VLAN 0 to HW filter on device eth0 worker1 # [ 11.201678] dhcpcd[636]: eth0: waiting for carrier worker2 # [ 11.216549] 8021q: adding VLAN 0 to HW filter on device eth1 worker1 # [ 11.206664] dhcpcd[636]: eth0: waiting for carrier worker1 # [ 11.209454] dhcpcd[636]: eth0: carrier acquired worker2 # [ 11.230664] 8021q: adding VLAN 0 to HW filter on device eth0 worker1 # [ 11.216226] systemd[1]: Finished Address configuration of eth1. worker2 # [ 11.221513] dhcpcd[641]: eth0: waiting for carrier worker2 # [ 11.225928] dhcpcd[641]: eth0: waiting for carrier worker1 # [ 11.223666] dhcpcd[636]: DUID 00:01:00:01:32:46:0d:ed:52:54:00:12:34:56 worker2 # [ 11.228172] dhcpcd[641]: eth0: carrier acquired worker2 # [ 11.231099] network-addresses-eth1-start[653]: adding address 192.168.1.4/24... done worker1 # [ 11.228209] dhcpcd[636]: eth0: IAID 00:12:34:56 worker1 # [ 11.228934] dhcpcd[636]: eth0: adding address fe80::5054:ff:fe12:3456 lb # [ 11.231266] postgresql-pre-start[677]: selecting default "max_connections" ... 100 worker2 # [ 11.244668] systemd-logind[502]: Watching system buttons on /dev/input/event0 (gpio-keys) worker2 # [ 11.249154] dhcpcd[641]: DUID 00:01:00:01:32:46:0d:ed:52:54:00:12:34:56 worker2 # [ 11.252172] network-addresses-eth1-start[653]: adding address 2001:db8:1::4/64... done worker2 # [ 11.253250] dhcpcd[641]: eth0: IAID 00:12:34:56 worker2 # [ 11.253879] dhcpcd[641]: eth0: adding address fe80::5054:ff:fe12:3456 worker2 # [ 11.264759] systemd[1]: Finished Address configuration of eth1. worker1 # [ 11.289677] mousedev: PS/2 mouse device common for all mice worker2 # [ 11.314606] mousedev: PS/2 mouse device common for all mice worker1 # [ 11.304569] systemd-logind[503]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) client # [ 11.331002] dhcpcd[579]: dhcpcd-10.3.2 starting client # [ 11.340891] dhcpcd[585]: dev: loaded udev lb # [ 11.359771] postgresql-pre-start[677]: selecting default "shared_buffers" ... 128MB client # [ 11.385770] 8021q: 802.1Q VLAN Support v1.8 lb # [ 11.400549] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker2 # [ 11.404130] systemd-logind[502]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) lb # [ 11.436434] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. lb # [ 11.444437] systemd[1]: Started backdoor.service. client # [ 11.495127] cfg80211: Loading compiled-in X.509 certificates for regulatory database lb # connecting to host... client # [ 11.535416] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' client # [ 11.535874] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' client # [ 11.538936] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 client # [ 11.539248] cfg80211: failed to load regulatory.db lb # [ 11.564586] 8021q: 802.1Q VLAN Support v1.8 client # [ 11.625641] 8021q: adding VLAN 0 to HW filter on device eth0 client # [ 11.614711] dhcpcd[585]: eth0: waiting for carrier client # [ 11.616095] dhcpcd[585]: libudev: received NULL device client # [ 11.619194] dhcpcd[585]: libudev: received NULL device client # [ 11.623021] dhcpcd[585]: eth0: carrier acquired lb # [ 11.620616] (udev-worker)[690]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. lb # [ 11.622727] (udev-worker)[690]: Network interface NamePolicy= disabled on kernel command line. client # [ 11.633122] dhcpcd[585]: DUID 00:01:00:01:32:46:0d:ee:52:54:00:12:34:56 client # [ 11.634044] dhcpcd[585]: eth0: IAID 00:12:34:56 client # [ 11.634672] dhcpcd[585]: eth0: adding address fe80::5054:ff:fe12:3456 lb # [ 11.641152] (udev-worker)[694]: Network interface NamePolicy= disabled on kernel command line. client # [ 11.657871] systemd[1]: Finished nftables firewall. client # [ 11.660111] systemd[1]: Reached target Preparation for Network. client # [ 11.664134] systemd[1]: Starting Address configuration of eth1... client # [ 11.666902] systemd[1]: Starting Extra networking commands.... lb # [ 11.695835] cfg80211: Loading compiled-in X.509 certificates for regulatory database lb # [ 11.720855] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' lb # [ 11.721309] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' lb # [ 11.724566] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 lb # [ 11.724868] cfg80211: failed to load regulatory.db client # [ 11.789566] 8021q: adding VLAN 0 to HW filter on device eth1 client # [ 11.805811] network-addresses-eth1-start[611]: adding address 192.168.1.1/24... done client # [ 11.852795] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 client # [ 11.844937] network-addresses-eth1-start[611]: adding address 2001:db8:1::1/64... done lb # [ 11.851953] systemd-vconsole-setup[696]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 11.860100] systemd[1]: Finished Virtual Console Setup. client # [ 11.873449] systemd[1]: Finished Address configuration of eth1. lb # [ 11.872194] systemd[1]: Condition check resulted in Virtio network device being skipped. lb # [ 11.881022] systemd[1]: Starting Address configuration of eth1... worker2 # [ 11.913694] dhcpcd[641]: eth0: soliciting a DHCP lease worker2 # [ 11.920736] dhcpcd[641]: eth0: offered 10.0.2.15 from 10.0.2.2 worker2 # [ 11.928224] dhcpcd[641]: eth0: probing address 10.0.2.15/24 lb # [ 11.958102] 8021q: adding VLAN 0 to HW filter on device eth0 lb # [ 11.947165] dhcpcd[617]: eth0: waiting for carrier lb # [ 11.950252] dhcpcd[617]: eth0: carrier acquired lb # [ 11.965653] dhcpcd[617]: DUID 00:01:00:01:32:46:0d:ee:52:54:00:12:34:56 lb # [ 11.966613] dhcpcd[617]: eth0: IAID 00:12:34:56 lb # [ 11.967241] dhcpcd[617]: eth0: adding address fe80::5054:ff:fe12:3456 client # [ 11.984609] systemd[1]: Finished Extra networking commands.. client # [ 11.987742] systemd[1]: Reached target Network. lb # [ 11.986411] systemd-logind[480]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 11.993339] systemd[1]: Starting Permit User Sessions... lb # [ 12.004528] 8021q: adding VLAN 0 to HW filter on device eth1 lb # [ 12.012509] network-addresses-eth1-start[718]: adding address 192.168.1.2/24... done worker1 # [ 12.035628] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 lb # [ 12.027401] network-addresses-eth1-start[718]: adding address 2001:db8:1::2/64... done lb # [ 12.050183] systemd[1]: Finished Address configuration of eth1. client # [ 12.078095] systemd[1]: Finished Permit User Sessions. worker2 # [ 12.090606] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 client # [ 12.082085] systemd[1]: Started Getty on tty1. client # [ 12.083559] systemd[1]: Reached target Login Prompts. lb # [ 12.113085] mousedev: PS/2 mouse device common for all mice client: (finished: waiting for unit network-addresses-eth1.service, in 12.57 seconds) client: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; } client # [ 12.173196] dhcpcd[585]: eth0: soliciting a DHCP lease client # [ 12.180532] dhcpcd[585]: eth0: offered 10.0.2.15 from 10.0.2.2 client # [ 12.188230] dhcpcd[585]: eth0: probing address 10.0.2.15/24 client: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; }, in 0.04 seconds) lb: waiting for unit network-addresses-eth1.service lb: waiting for the VM to finish booting lb: Guest shell says: b'Spawning backdoor root shell...\n' lb: connected to guest root shell lb: (connecting took 0.00 seconds) lb: (finished: waiting for the VM to finish booting, in 0.00 seconds) lb # [ 12.233132] systemd-logind[480]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) lb: (finished: waiting for unit network-addresses-eth1.service, in 0.08 seconds) lb: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; } client # [ 12.278892] systemd[1]: Starting Virtual Console Setup... client # [ 12.293821] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 12.298775] systemd[1]: Stopped Virtual Console Setup. client # [ 12.306319] systemd[1]: Starting Virtual Console Setup... worker1 # [ 12.318672] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. lb: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; }, in 0.05 seconds) worker1: waiting for unit network-addresses-eth1.service worker1: waiting for the VM to finish booting worker1: Guest shell says: b'Spawning backdoor root shell...\n' worker1: connected to guest root shell worker1: (connecting took 0.00 seconds) worker1: (finished: waiting for the VM to finish booting, in 0.00 seconds) client # [ 12.328690] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker1 # [ 12.325148] systemd[1]: Starting Virtual Console Setup... worker1 # [ 12.352651] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 12.361561] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 12.364270] systemd[1]: Starting Virtual Console Setup... client # [ 12.374574] systemd-logind[484]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker2 # [ 12.386513] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker2 # [ 12.409295] systemd[1]: Starting Virtual Console Setup... worker2 # [ 12.426049] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1: (finished: waiting for unit network-addresses-eth1.service, in 0.11 seconds) worker1: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; } worker2 # [ 12.432516] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 12.439437] systemd[1]: Starting Virtual Console Setup... client # [ 12.456884] systemd-vconsole-setup[698]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 12.460583] systemd[1]: Finished Virtual Console Setup. worker1 # [ 12.465800] systemd-logind[503]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker1: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; }, in 0.05 seconds) worker2: waiting for unit network-addresses-eth1.service worker2: waiting for the VM to finish booting worker2: Guest shell says: b'Spawning backdoor root shell...\n' worker2: connected to guest root shell worker2: (connecting took 0.00 seconds) worker2: (finished: waiting for the VM to finish booting, in 0.00 seconds) worker2 # [ 12.506891] systemd-logind[502]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker1 # [ 12.514737] dhcpcd[636]: eth0: soliciting an IPv6 router worker1 # [ 12.516101] dhcpcd[636]: eth0: Router Advertisement from fe80::2 worker1 # [ 12.517179] dhcpcd[636]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker1 # [ 12.518145] dhcpcd[636]: eth0: adding route to fec0::/64 worker1 # [ 12.518879] dhcpcd[636]: eth0: adding default route via fe80::2 worker2: (finished: waiting for unit network-addresses-eth1.service, in 0.06 seconds) worker2: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; } worker1 # [ 12.549166] systemd-vconsole-setup[695]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 12.553516] systemd[1]: Finished Virtual Console Setup. worker2: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; }, in 0.03 seconds) lb: waiting for unit niks3.service worker2 # [ 12.617845] systemd-vconsole-setup[694]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 12.621445] systemd[1]: Finished Virtual Console Setup. worker1 # [ 12.762605] dhcpcd[636]: eth0: soliciting a DHCP lease worker1 # [ 12.765961] dhcpcd[636]: eth0: offered 10.0.2.15 from 10.0.2.2 worker1 # [ 12.768817] dhcpcd[636]: eth0: probing address 10.0.2.15/24 worker2 # [ 13.250099] dhcpcd[641]: eth0: soliciting an IPv6 router worker2 # [ 13.253396] dhcpcd[641]: eth0: Router Advertisement from fe80::2 worker2 # [ 13.255897] dhcpcd[641]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker2 # [ 13.259142] dhcpcd[641]: eth0: adding route to fec0::/64 worker2 # [ 13.261533] dhcpcd[641]: eth0: adding default route via fe80::2 client # [ 13.449038] dhcpcd[585]: eth0: soliciting an IPv6 router client # [ 13.451364] dhcpcd[585]: eth0: Router Advertisement from fe80::2 client # [ 13.454470] dhcpcd[585]: eth0: adding address fec0::5054:ff:fe12:3456/64 client # [ 13.457328] dhcpcd[585]: eth0: adding route to fec0::/64 client # [ 13.459504] dhcpcd[585]: eth0: adding default route via fe80::2 lb # [ 13.467803] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. lb # [ 13.469484] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 13.470767] systemd[1]: Stopped Virtual Console Setup. lb # [ 13.475679] systemd[1]: Stopping Virtual Console Setup... lb # [ 13.480369] systemd[1]: Starting Virtual Console Setup... lb # [ 13.504085] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 13.507310] systemd[1]: Stopped Virtual Console Setup. lb # [ 13.513677] systemd[1]: Starting Virtual Console Setup... lb # [ 13.539910] systemd-logind[480]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) lb # [ 13.852910] dhcpcd[617]: eth0: soliciting an IPv6 router lb # [ 13.856073] dhcpcd[617]: eth0: Router Advertisement from fe80::2 lb # [ 13.856901] dhcpcd[617]: eth0: adding address fec0::5054:ff:fe12:3456/64 lb # [ 13.857794] dhcpcd[617]: eth0: adding route to fec0::/64 lb # [ 13.860108] dhcpcd[617]: eth0: adding default route via fe80::2 lb # [ 13.876264] dhcpcd[617]: eth0: soliciting a DHCP lease lb # [ 13.880349] dhcpcd[617]: eth0: offered 10.0.2.15 from 10.0.2.2 lb # [ 13.884190] dhcpcd[617]: eth0: probing address 10.0.2.15/24 lb # [ 13.892592] postgresql-pre-start[677]: selecting default time zone ... UTC lb # [ 13.895843] postgresql-pre-start[677]: creating configuration files ... ok lb # [ 13.999696] systemd-vconsole-setup[759]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 14.004387] systemd[1]: Finished Virtual Console Setup. lb # [ 14.135651] postgresql-pre-start[677]: running bootstrap script ... ok lb # [ 14.627591] postgresql-pre-start[677]: performing post-bootstrap initialization ... ok lb # [ 14.777464] postgresql-pre-start[677]: syncing data to disk ... ok lb # [ 14.778552] postgresql-pre-start[677]: initdb: warning: enabling "trust" authentication for local connections lb # [ 14.781026] postgresql-pre-start[677]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. lb # [ 14.783030] postgresql-pre-start[677]: Success. You can now start the database server using: lb # [ 14.784168] postgresql-pre-start[677]: pg_ctl -D /var/lib/postgresql/18 -l logfile start lb # [ 14.911299] postgres[778]: [778] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit lb # [ 14.914208] postgres[778]: [778] LOG: listening on IPv6 address "::1", port 5432 lb # [ 14.915356] postgres[778]: [778] LOG: listening on IPv4 address "127.0.0.1", port 5432 lb # [ 14.917295] postgres[778]: [778] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432" lb # [ 14.927807] postgres[796]: [796] LOG: database system was shut down at 2026-09-23 04:11:29 GMT lb # [ 14.934296] postgres[778]: [778] LOG: database system is ready to accept connections lb # [ 14.938549] systemd[1]: Started PostgreSQL Server. lb # [ 14.946477] systemd[1]: Starting PostgreSQL Setup Scripts... lb # [ 15.137924] postgresql-setup-start[807]: CREATE DATABASE lb # [ 15.191080] postgresql-setup-start[813]: CREATE ROLE lb # [ 15.210981] postgresql-setup-start[818]: ALTER DATABASE lb # [ 15.217259] systemd[1]: Finished PostgreSQL Setup Scripts. lb # [ 15.218879] systemd[1]: Reached target PostgreSQL. worker2 # [ 16.489635] dhcpcd[641]: eth0: leased 10.0.2.15 for 86400 seconds worker2 # [ 16.492738] dhcpcd[641]: eth0: adding route to 10.0.2.0/24 worker2 # [ 16.495034] dhcpcd[641]: eth0: adding default route via 10.0.2.2 worker2 # [ 16.628325] systemd[1]: Started DHCP Client. worker2 # [ 16.630407] systemd[1]: Reached target Multi-User System. worker2 # [ 16.631353] systemd[1]: Startup finished in 996ms (kernel) + 4.956s (initrd) + 10.677s (userspace) = 16.630s. worker1 # [ 17.201049] dhcpcd[636]: eth0: leased 10.0.2.15 for 86400 seconds worker1 # [ 17.204334] dhcpcd[636]: eth0: adding route to 10.0.2.0/24 client # [ 17.209721] dhcpcd[585]: eth0: leased 10.0.2.15 for 86400 seconds worker1 # [ 17.206622] dhcpcd[636]: eth0: adding default route via 10.0.2.2 client # [ 17.213469] dhcpcd[585]: eth0: adding route to 10.0.2.0/24 client # [ 17.215774] dhcpcd[585]: eth0: adding default route via 10.0.2.2 client # [ 17.347478] systemd[1]: Started DHCP Client. client # [ 17.350047] systemd[1]: Reached target Multi-User System. worker1 # [ 17.345534] systemd[1]: Started DHCP Client. client # [ 17.351022] systemd[1]: Startup finished in 966ms (kernel) + 4.979s (initrd) + 11.403s (userspace) = 17.349s. worker1 # [ 17.347865] systemd[1]: Reached target Multi-User System. worker1 # [ 17.349422] systemd[1]: Startup finished in 968ms (kernel) + 4.998s (initrd) + 11.380s (userspace) = 17.347s. lb # [ 18.999110] dhcpcd[617]: eth0: leased 10.0.2.15 for 86400 seconds lb # [ 19.002564] dhcpcd[617]: eth0: adding route to 10.0.2.0/24 lb # [ 19.007000] dhcpcd[617]: eth0: adding default route via 10.0.2.2 lb # [ 19.150669] systemd[1]: Started DHCP Client. lb # [ 19.152925] systemd[1]: Reached target Network is Online. lb # [ 19.157442] systemd[1]: Started Envoy reverse proxy. lb # [ 19.165187] systemd[1]: Started mock-oidc.service. lb # [ 19.625122] mock-oidc-server[907]: Mock OIDC Server running lb # [ 19.627202] mock-oidc-server[907]: OIDC Address: 192.168.1.2:8080 lb # [ 19.628384] mock-oidc-server[907]: Issue Address: 0.0.0.0:8081 lb # [ 19.629463] mock-oidc-server[907]: Issuer: http://192.168.1.2:8080/oidc lb # [ 19.630611] mock-oidc-server[907]: JWKS: http://192.168.1.2:8080/oidc/.well-known/jwks.json lb # [ 19.631972] mock-oidc-server[907]: Discovery: http://192.168.1.2:8080/oidc/.well-known/openid-configuration lb # [ 19.633623] mock-oidc-server[907]: Issue tokens: http://0.0.0.0:8081/issue?sub=... lb # [ 19.669201] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:467] initializing epoch 0 (base id=0, hot restart version=11.120) lb # [ 19.671053] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:469] statically linked extensions: lb # [ 19.672738] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.health_check.event_sinks: envoy.health_check.event_sink.file lb # [ 19.674680] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator lb # [ 19.677519] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest lb # [ 19.683679] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.filters.listener: envoy.filters.listener.dynamic_modules, envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.set_filter_state, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector lb # [ 19.713702] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.matching.action: envoy.matching.action.transport_socket.name, envoy.matching.actions.format_string, filter-chain-name lb # [ 19.717106] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 19.719447] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_mappers: envoy.tls.certificate_mappers.static_name, envoy.tls.upstream_certificate_mappers.filter_state_override lb # [ 19.722997] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.http.forward_client_cert_details: forward_client_cert lb # [ 19.725421] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor lb # [ 19.728392] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.filters.upstream_network: envoy.filters.upstream_network.reverse_tunnel_lifecycle lb # [ 19.730405] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.quic.connection_id_generator: envoy.quic.connection_id_generator.quic_lb, envoy.quic.deterministic_connection_id_generator lb # [ 19.733098] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.http.injected_credentials: envoy.http.injected_credentials.generic, envoy.http.injected_credentials.oauth2 lb # [ 19.736572] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.rbac.principals: envoy.rbac.principals.mtls_authenticated lb # [ 19.738346] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.ext_authz, envoy.filters.udp.session.http_capsule lb # [ 19.741376] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.geoip_providers: envoy.geoip_providers.maxmind lb # [ 19.743076] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.credential_injector, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec, envoy.filters.http.upstream_rbac, envoy.filters.http.wasm lb # [ 19.751930] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.generic_proxy.filters: envoy.filters.generic.router lb # [ 19.753932] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.upstream_options: envoy.extensions.upstreams.http.reverse_tunnel.v3.ReverseTunnelUpstreamCodecOptions, envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options lb # [ 19.758748] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector, envoy.upstream.local_address_selector.filter_state_override lb # [ 19.761871] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.matching.stat_matching_data.input: stat_full_name_match_input lb # [ 19.763700] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.bootstrap: envoy.bootstrap.dynamic_modules, envoy.bootstrap.internal_listener, envoy.bootstrap.reverse_tunnel.downstream_socket_interface, envoy.bootstrap.reverse_tunnel.upstream_socket_interface, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface, envoy.extensions.network.socket_interface.sockmap lb # [ 19.768770] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.matching.transport_socket.input: envoy.matching.inputs.endpoint_metadata, envoy.matching.inputs.locality_metadata, envoy.matching.inputs.transport_socket_filter_state lb # [ 19.771600] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso lb # [ 19.773793] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher lb # [ 19.775583] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.resource_monitors: envoy.resource_monitors.cgroup_memory, envoy.resource_monitors.cpu_utilization, envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource lb # [ 19.778865] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter lb # [ 19.780889] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.rds_factory: envoy.rds_factory.default lb # [ 19.782489] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter lb # [ 19.784498] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.regex_engines: envoy.regex_engines.google_re2 lb # [ 19.786157] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts lb # [ 19.789295] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme lb # [ 19.792514] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.matching.input_matchers: envoy.matching.input_matchers.generic_request_matcher, envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.dynamic_modules, envoy.matching.matchers.ip, envoy.matching.matchers.metadata_matcher, envoy.matching.matchers.runtime_fraction lb # [ 19.797181] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.health_checkers: envoy.health_checkers.dynamic_modules, envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift lb # [ 19.800421] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation lb # [ 19.802356] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.retry_priorities: envoy.retry_priorities.previous_priorities lb # [ 19.804207] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.access_loggers: envoy.access_loggers.dynamic_modules, envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stats, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log lb # [ 19.811120] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable lb # [ 19.813518] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.quic.client_packet_writer: envoy.quic.packet_writer.default lb # [ 19.815533] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.srds_factory: envoy.srds_factory.default lb # [ 19.817561] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel, envoy.access_loggers.extension_filters.process_ratelimit lb # [ 19.820404] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.fault, envoy.filters.http.a2a, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.ai_protocol_manager, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.api_key_auth, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.bandwidth_share, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cache_v2, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_server, envoy.filters.http.file_system_buffer, envoy.filters.http.filter_chain, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_reverse_transcoder, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.mcp, envoy.filters.http.mcp_json_rest_bridge, envoy.filters.http.mcp_router, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.proto_api_scrubber, envoy.filters.http.proto_message_extraction, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.sse_to_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.thrift_to_metadata, envoy.filters.http.transform, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router lb # [ 19.850464] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.hashable_string, envoy.network.application_protocols, envoy.network.ip, envoy.network.network_namespace, envoy.network.transport_socket.http_11_proxy.info, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.ratelimit.hits_addend, envoy.router.debug_config, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port lb # [ 19.859608] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr lb # [ 19.862544] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy lb # [ 19.865741] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche lb # [ 19.867824] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.envelope, envoy.http.stateful_session.header lb # [ 19.870720] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.quic.server_preferred_address: quic.server_preferred_address.datasource, quic.server_preferred_address.fixed lb # [ 19.873635] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.http.cache_v2: envoy.extensions.http.cache_v2.file_system_http_cache, envoy.extensions.http.cache_v2.simple lb # [ 19.878455] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory lb # [ 19.881575] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.route_config_update_requester: envoy.route_config_update_requester.default lb # [ 19.883517] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] network.connection.client: default, envoy_internal worker2 # [ 19.884692] systemd[1]: Reached target Network is Online. lb # [ 19.886407] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.protocols: dubbo worker1 # [ 19.886475] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 19.889338] envoy[906]: [2026-09-23 04:11:34.283][906][info][main] [source/server/server.cc:471] envoy.upstreams: envoy.filters.connection_pools.tcp.generic lb # [ 19.891329] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.built_in_formatters: envoy.built_in_formatters.cel, envoy.built_in_formatters.http.default, envoy.built_in_formatters.metadata, envoy.built_in_formatters.stream_info.default, envoy.built_in_formatters.xfcc_value worker2 # [ 19.894516] systemd[1]: Starting scheduler-token.service... lb # [ 19.894990] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo, envoy.network.dns_resolver.hickory lb # [ 19.897708] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls lb # [ 19.903751] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.string_matcher: envoy.string_matcher.lua lb # [ 19.905494] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 19.907979] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.dynamic_metadata, envoy.matching.inputs.dynamic_module_data_input, envoy.matching.inputs.filter_state, envoy.matching.inputs.local_reply, envoy.matching.inputs.network_namespace, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params lb # [ 19.918833] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.matching.stat_tag_matching_data.input: stat_tag_value_input lb # [ 19.920636] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.grpc_credentials: envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata lb # [ 19.922710] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.thrift_proxy.transports: auto, framed, header, unframed lb # [ 19.924465] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.tracers: envoy.tracers.datadog, envoy.tracers.dynamic_modules, envoy.tracers.fluentd, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin lb # [ 19.927561] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.quic.connection_debug_visitor: envoy.quic.connection_debug_visitor.basic, envoy.quic.connection_debug_visitor.quic_stats lb # [ 19.930123] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.clusters: envoy.cluster.dns, envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.composite, envoy.clusters.dynamic_forward_proxy, envoy.clusters.dynamic_modules, envoy.clusters.mcp_multicluster, envoy.clusters.redis, envoy.clusters.reverse_connection lb # [ 19.935533] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.matching.generic_proxy_request_input.input: envoy.matching.generic_proxy.input.host, envoy.matching.generic_proxy.input.method, envoy.matching.generic_proxy.input.path, envoy.matching.generic_proxy.input.property, envoy.matching.generic_proxy.input.request, envoy.matching.generic_proxy.input.service lb # [ 19.941328] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.load_balancing_policies: envoy.load_balancing_policies.client_side_weighted_round_robin, envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.dynamic_modules, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.load_aware_locality, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.override_host, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset, envoy.load_balancing_policies.wrr_locality lb # [ 19.948591] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.common.key_value: envoy.key_value.file_based lb # [ 19.950228] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.request_id: envoy.request_id.uuid lb # [ 19.951772] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.generic_proxy.codecs: envoy.generic_proxy.codecs.dubbo, envoy.generic_proxy.codecs.http1 lb # [ 19.954044] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.http.ext_proc.response_processors: envoy.extensions.http.ext_proc.save_processing_response lb # [ 19.956165] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.route.early_data_policy: envoy.route.early_data_policy.default lb # [ 19.957986] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8 lb # [ 19.959780] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain lb # [ 19.961663] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.dynamic_modules, envoy.tls.cert_validator.spiffe lb # [ 19.964201] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router lb # [ 19.967065] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router lb # [ 19.968810] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor lb # [ 19.971304] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.content_parsers: envoy.content_parsers.json lb # [ 19.973012] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.dynamic_modules, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd lb # [ 19.978000] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.dynamic_modules, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.ext_proc, envoy.filters.network.generic_proxy, envoy.filters.network.geoip, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.reverse_tunnel, envoy.filters.network.reverse_tunnel_drain_aware_http_connection_manager, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_bandwidth_limit, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy lb # [ 19.991465] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port lb # [ 19.994145] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.serializers: dubbo.hessian2 lb # [ 19.998453] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation lb # [ 20.000915] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple lb # [ 20.003114] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.http.header_validators: envoy.http.header_validators.envoy_default lb # [ 20.005119] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.ssl.server_context_factory: envoy.ssl.server_context_factory.default worker2 # [ 20.008549] systemd[1]: scheduler-token.service: Deactivated successfully. lb # [ 20.007020] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 20.009647] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.network_namespace, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san worker2 # [ 20.016651] systemd[1]: Finished scheduler-token.service. lb # [ 20.016666] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.tls.certificate_mappers: envoy.tls.certificate_mappers.sni, envoy.tls.certificate_mappers.static_name lb # [ 20.018896] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.connection_handler: envoy.connection_handler.default lb # [ 20.020719] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] quic.http_server_connection: quic.http_server_connection.default lb # [ 20.022520] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action lb # [ 20.024536] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.resolvers: envoy.ip, envoy.resolvers.reverse_connection worker2 # [ 20.023612] systemd[1]: scheduler-token.service: Consumed 65ms CPU time over 117ms wall clock time, 3.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb # [ 20.026303] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.http.ext_proc.processing_request_modifiers: envoy.extensions.http.ext_proc.mapped_attribute_builder lb # [ 20.028529] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment, envoy.tracers.opentelemetry.resource_detectors.static_config lb # [ 20.032257] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.tls.certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 20.034746] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.formatter: envoy.formatter.cel, envoy.formatter.dynamic_modules, envoy.formatter.file_content, envoy.formatter.generic_secret, envoy.formatter.metadata, envoy.formatter.req_without_query worker2 # [ 20.041400] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 20.038221] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua, envoy.router.cluster_specifier_plugin.matcher lb # [ 20.040905] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.dynamic_modules, envoy.filters.udp_listener.udp_proxy lb # [ 20.043548] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff lb # [ 20.045988] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.cel, envoy.tracers.opentelemetry.samplers.dynatrace, envoy.tracers.opentelemetry.samplers.parent_based, envoy.tracers.opentelemetry.samplers.trace_id_ratio_based lb # [ 20.050065] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case lb # [ 20.052286] envoy[906]: [2026-09-23 04:11:34.284][906][info][main] [source/server/server.cc:471] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls lb # [ 20.056783] envoy[906]: [2026-09-23 04:11:34.309][906][info][main] [source/server/server.cc:531] HTTP header map info: worker1 # [ 20.066740] nix-grpc-daemon[766]: ts=2026-09-23T04:11:34Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" lb # [ 20.058131] envoy[906]: [2026-09-23 04:11:34.309][906][info][main] [source/server/server.cc:534] request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-host,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-request-id lb # [ 20.073374] envoy[906]: [2026-09-23 04:11:34.309][906][info][main] [source/server/server.cc:534] request trailer map: 120 bytes: worker1 # [ 20.075384] nix-grpc-daemon[766]: ts=2026-09-23T04:11:34Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 20.077985] nix-grpc-daemon[766]: ts=2026-09-23T04:11:34Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 lb # [ 20.074825] envoy[906]: [2026-09-23 04:11:34.309][906][info][main] [source/server/server.cc:534] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id lb # [ 20.083712] envoy[906]: [2026-09-23 04:11:34.309][906][info][main] [source/server/server.cc:534] response trailer map: 144 bytes: grpc-message,grpc-status lb # [ 20.085545] envoy[906]: [2026-09-23 04:11:34.326][906][info][main] [source/server/server.cc:968] runtime: {} lb # [ 20.086762] envoy[906]: [2026-09-23 04:11:34.328][906][info][main] [source/server/server.cc:420] Raised file descriptor limits from 1024 to 524288. lb # [ 20.088411] envoy[906]: [2026-09-23 04:11:34.328][906][info][main] [source/server/server.cc:796] Starting admin HTTP server at 127.0.0.1:9901 worker1 # [ 20.086871] nix-grpc-daemon[766]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR lb # [ 20.090088] envoy[906]: [2026-09-23 04:11:34.329][906][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:9901 lb # [ 20.091592] envoy[906]: [2026-09-23 04:11:34.331][906][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration worker1 # [ 20.089862] nix-grpc-daemon[766]: E0000 00:00:1790136694.689215 766 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 20.093299] envoy[906]: [2026-09-23 04:11:34.331][906][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s) lb # [ 20.094818] envoy[906]: [2026-09-23 04:11:34.331][906][info][config] [source/server/configuration_impl.cc:138] loading 2 cluster(s) lb # [ 20.096385] envoy[906]: [2026-09-23 04:11:34.492][906][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s) worker1 # [ 20.095609] nix-grpc-daemon[766]: E0000 00:00:1790136694.697931 766 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb # [ 20.097885] envoy[906]: [2026-09-23 04:11:34.504][906][info][config] [source/server/configuration_impl.cc:164] loading stats configuration lb # [ 20.099430] envoy[906]: [2026-09-23 04:11:34.504][906][warning][main] [source/server/server.cc:1038] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor. lb # [ 20.102577] envoy[906]: [2026-09-23 04:11:34.504][906][info][main] [source/server/server.cc:1077] starting main dispatch loop worker1 # [ 20.116221] nix-grpc-daemon[766]: ts=2026-09-23T04:11:34Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 20.118186] systemd[1]: Started Nix worker-protocol over gRPC. lb # [ 20.124425] envoy[906]: [2026-09-23 04:11:34.739][906][info][runtime] [source/common/runtime/runtime_impl.cc:567] RTDS has finished initialization worker1 # [ 20.124065] nix-grpc-daemon[766]: E0000 00:00:1790136694.725783 808 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 20.126174] envoy[906]: [2026-09-23 04:11:34.739][906][info][upstream] [source/common/upstream/cluster_manager_impl.cc:250] cm init: all clusters initialized lb # [ 20.127907] envoy[906]: [2026-09-23 04:11:34.739][906][info][main] [source/server/server.cc:1057] all clusters initialized. initializing init manager lb # [ 20.132126] envoy[906]: [2026-09-23 04:11:34.746][906][info][config] [source/common/listener_manager/listener_manager_impl.cc:1067] all dependencies initialized. starting workers worker1 # [ 20.142419] nix-grpc-daemon[766]: E0000 00:00:1790136694.744770 808 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 20.148049] nix-grpc-daemon[766]: E0000 00:00:1790136694.747602 808 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 20.163739] nix-grpc-daemon[766]: E0000 00:00:1790136694.765583 808 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 20.174058] nix-grpc-daemon[766]: ts=2026-09-23T04:11:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.175744] nix-grpc-daemon[766]: ts=2026-09-23T04:11:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.182851] nix-grpc-daemon[771]: ts=2026-09-23T04:11:34Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 20.190800] nix-grpc-daemon[771]: ts=2026-09-23T04:11:34Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 20.192944] nix-grpc-daemon[771]: ts=2026-09-23T04:11:34Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 20.205896] nix-grpc-daemon[771]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 20.208097] nix-grpc-daemon[771]: E0000 00:00:1790136694.784836 771 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 20.213880] nix-grpc-daemon[771]: E0000 00:00:1790136694.792978 771 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 20.232912] nix-grpc-daemon[771]: ts=2026-09-23T04:11:34Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 20.236252] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 20.240618] nix-grpc-daemon[771]: E0000 00:00:1790136694.817090 814 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 20.259546] nix-grpc-daemon[771]: E0000 00:00:1790136694.838690 814 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 20.269967] nix-grpc-daemon[771]: E0000 00:00:1790136694.841448 814 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 20.281236] nix-grpc-daemon[771]: E0000 00:00:1790136694.859552 814 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 20.294150] nix-grpc-daemon[771]: ts=2026-09-23T04:11:34Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.295843] nix-grpc-daemon[771]: ts=2026-09-23T04:11:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.376836] nix-grpc-daemon[766]: ts=2026-09-23T04:11:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.382033] nix-grpc-daemon[766]: ts=2026-09-23T04:11:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.497261] nix-grpc-daemon[771]: ts=2026-09-23T04:11:35Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.502252] nix-grpc-daemon[771]: ts=2026-09-23T04:11:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.582759] nix-grpc-daemon[766]: ts=2026-09-23T04:11:35Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.587883] nix-grpc-daemon[766]: ts=2026-09-23T04:11:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.703372] nix-grpc-daemon[771]: ts=2026-09-23T04:11:35Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.708650] nix-grpc-daemon[771]: ts=2026-09-23T04:11:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.788660] nix-grpc-daemon[766]: ts=2026-09-23T04:11:35Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.793738] nix-grpc-daemon[766]: ts=2026-09-23T04:11:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.909567] nix-grpc-daemon[771]: ts=2026-09-23T04:11:35Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.914585] nix-grpc-daemon[771]: ts=2026-09-23T04:11:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.995966] nix-grpc-daemon[766]: ts=2026-09-23T04:11:35Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.999965] nix-grpc-daemon[766]: ts=2026-09-23T04:11:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.115694] nix-grpc-daemon[771]: ts=2026-09-23T04:11:35Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.120922] nix-grpc-daemon[771]: ts=2026-09-23T04:11:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.198574] nix-grpc-daemon[766]: ts=2026-09-23T04:11:35Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.202670] nix-grpc-daemon[766]: ts=2026-09-23T04:11:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.323411] nix-grpc-daemon[771]: ts=2026-09-23T04:11:35Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.327576] nix-grpc-daemon[771]: ts=2026-09-23T04:11:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.399140] nix-grpc-daemon[766]: ts=2026-09-23T04:11:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.404834] nix-grpc-daemon[766]: ts=2026-09-23T04:11:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.523901] nix-grpc-daemon[771]: ts=2026-09-23T04:11:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.529016] nix-grpc-daemon[771]: ts=2026-09-23T04:11:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.605607] nix-grpc-daemon[766]: ts=2026-09-23T04:11:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.610600] nix-grpc-daemon[766]: ts=2026-09-23T04:11:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.729986] nix-grpc-daemon[771]: ts=2026-09-23T04:11:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.735069] nix-grpc-daemon[771]: ts=2026-09-23T04:11:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.811459] nix-grpc-daemon[766]: ts=2026-09-23T04:11:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.816644] nix-grpc-daemon[766]: ts=2026-09-23T04:11:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.936141] nix-grpc-daemon[771]: ts=2026-09-23T04:11:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.941091] nix-grpc-daemon[771]: ts=2026-09-23T04:11:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.017495] nix-grpc-daemon[766]: ts=2026-09-23T04:11:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.022496] nix-grpc-daemon[766]: ts=2026-09-23T04:11:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.142133] nix-grpc-daemon[771]: ts=2026-09-23T04:11:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.147230] nix-grpc-daemon[771]: ts=2026-09-23T04:11:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 22.183647] rustfs-bucket-start[951]: mb s3://farm lb # [ 22.198817] systemd[1]: Finished rustfs-bucket.service. lb # [ 22.209955] systemd[1]: Starting niks3 server... worker1 # [ 22.225271] nix-grpc-daemon[766]: ts=2026-09-23T04:11:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.229321] nix-grpc-daemon[766]: ts=2026-09-23T04:11:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 22.348942] postgres[967]: [967] ERROR: relation "goose_db_version" does not exist at character 36 worker2 # [ 22.349898] nix-grpc-daemon[771]: ts=2026-09-23T04:11:36Z level=info event=scheduler_connected addr=lb:50051 lb # [ 22.350184] postgres[967]: [967] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC worker2 # [ 22.354008] nix-grpc-daemon[771]: ts=2026-09-23T04:11:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 22.375437] niks3-server[962]: 2026/09/23 04:11:36 OK 20241026095416_initial_model.sql (14.51ms) lb # [ 22.378418] niks3-server[962]: 2026/09/23 04:11:36 OK 20251210153512_drop_unused_gin_index.sql (3.24ms) lb # [ 22.383788] niks3-server[962]: 2026/09/23 04:11:36 OK 20251218171726_add_pins.sql (5.31ms) lb # [ 22.388210] niks3-server[962]: 2026/09/23 04:11:37 OK 20260628120000_add_object_size_and_stats.sql (4.39ms) lb # [ 22.393383] niks3-server[962]: 2026/09/23 04:11:37 OK 20260905000000_add_claims.sql (5.14ms) lb # [ 22.396970] niks3-server[962]: 2026/09/23 04:11:37 OK 20260920000000_drop_claims.sql (3.49ms) lb # [ 22.398754] niks3-server[962]: 2026/09/23 04:11:37 goose: successfully migrated database to version: 20260920000000 lb # [ 22.402028] niks3-server[962]: 2026/09/23 04:11:37 OK 1_commit_pending_closure.sql (5.03ms) lb # [ 22.404492] niks3-server[962]: 2026/09/23 04:11:37 OK 2_object_stats_trigger.sql (2.34ms) lb # [ 22.406345] niks3-server[962]: 2026/09/23 04:11:37 goose: up to current file version: 2 lb # [ 22.408097] niks3-server[962]: 2026/09/23 04:11:37 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key worker1 # [ 22.425440] nix-grpc-daemon[766]: ts=2026-09-23T04:11:37Z level=info event=scheduler_connected addr=lb:50051 lb # [ 22.427902] niks3-server[962]: 2026/09/23 04:11:37 INFO Read proxy enabled — serving cache objects from S3 lb # [ 22.431689] niks3-server[962]: 2026/09/23 04:11:37 INFO Using socket-activated listener address=0.0.0.0:5751 worker1 # [ 22.430540] nix-grpc-daemon[766]: ts=2026-09-23T04:11:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 22.436202] niks3-server[962]: 2026/09/23 04:11:37 INFO systemd watchdog enabled interval=15s lb # [ 22.438916] niks3-server[962]: 2026/09/23 04:11:37 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 22.440648] systemd[1]: Started niks3 server. lb # [ 22.441597] systemd[1]: Reached target Multi-User System. lb # [ 22.442515] systemd[1]: Startup finished in 957ms (kernel) + 4.260s (initrd) + 17.219s (userspace) = 22.438s. worker2 # [ 22.550060] nix-grpc-daemon[771]: ts=2026-09-23T04:11:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.555082] nix-grpc-daemon[771]: ts=2026-09-23T04:11:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.631407] nix-grpc-daemon[766]: ts=2026-09-23T04:11:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.636547] nix-grpc-daemon[766]: ts=2026-09-23T04:11:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.756088] nix-grpc-daemon[771]: ts=2026-09-23T04:11:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.760940] nix-grpc-daemon[771]: ts=2026-09-23T04:11:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: waiting for unit niks3.service, in 10.21 seconds) lb: waiting for TCP port 5751 on localhost lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded! lb: (finished: waiting for TCP port 5751 on localhost, in 0.07 seconds) lb: waiting for unit envoy.service worker1 # [ 22.837321] nix-grpc-daemon[766]: ts=2026-09-23T04:11:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.842274] nix-grpc-daemon[766]: ts=2026-09-23T04:11:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: waiting for unit envoy.service, in 0.05 seconds) lb: waiting for TCP port 50051 on localhost lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded! lb # [ 22.913411] envoy[906]: conn peer=[::1]:46738 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=3 lb: (finished: waiting for TCP port 50051 on localhost, in 0.03 seconds) worker2 # [ 22.961949] nix-grpc-daemon[771]: ts=2026-09-23T04:11:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.966976] nix-grpc-daemon[771]: ts=2026-09-23T04:11:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.043250] nix-grpc-daemon[766]: ts=2026-09-23T04:11:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.048954] nix-grpc-daemon[766]: ts=2026-09-23T04:11:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.076910] systemd[1]: Starting scheduler-token.service... worker2 # [ 23.152619] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 23.160470] systemd[1]: Finished scheduler-token.service. worker2 # [ 23.162005] systemd[1]: scheduler-token.service: Consumed 45ms CPU time over 82ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 23.169259] nix-grpc-daemon[771]: ts=2026-09-23T04:11:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.172287] nix-grpc-daemon[771]: ts=2026-09-23T04:11:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.08 seconds) subtest: one node schedules and both builders hold a WorkerSession on it lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 23.251270] nix-grpc-daemon[766]: ts=2026-09-23T04:11:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.255391] nix-grpc-daemon[766]: ts=2026-09-23T04:11:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2 # [ 23.374975] nix-grpc-daemon[771]: ts=2026-09-23T04:11:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.379137] nix-grpc-daemon[771]: ts=2026-09-23T04:11:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.451886] nix-grpc-daemon[766]: ts=2026-09-23T04:11:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.457037] nix-grpc-daemon[766]: ts=2026-09-23T04:11:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.575713] nix-grpc-daemon[771]: ts=2026-09-23T04:11:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.580993] nix-grpc-daemon[771]: ts=2026-09-23T04:11:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.657511] nix-grpc-daemon[766]: ts=2026-09-23T04:11:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.662478] nix-grpc-daemon[766]: ts=2026-09-23T04:11:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.781854] nix-grpc-daemon[771]: ts=2026-09-23T04:11:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.786840] nix-grpc-daemon[771]: ts=2026-09-23T04:11:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.863302] nix-grpc-daemon[766]: ts=2026-09-23T04:11:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.868371] nix-grpc-daemon[766]: ts=2026-09-23T04:11:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.987788] nix-grpc-daemon[771]: ts=2026-09-23T04:11:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.992924] nix-grpc-daemon[771]: ts=2026-09-23T04:11:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.069066] nix-grpc-daemon[766]: ts=2026-09-23T04:11:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.074099] nix-grpc-daemon[766]: ts=2026-09-23T04:11:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.193981] nix-grpc-daemon[771]: ts=2026-09-23T04:11:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.198982] nix-grpc-daemon[771]: ts=2026-09-23T04:11:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.276518] nix-grpc-daemon[766]: ts=2026-09-23T04:11:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.280532] nix-grpc-daemon[766]: ts=2026-09-23T04:11:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 24.401289] nix-grpc-daemon[771]: ts=2026-09-23T04:11:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.405870] nix-grpc-daemon[771]: ts=2026-09-23T04:11:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.476946] nix-grpc-daemon[766]: ts=2026-09-23T04:11:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.482012] nix-grpc-daemon[766]: ts=2026-09-23T04:11:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.606792] nix-grpc-daemon[771]: ts=2026-09-23T04:11:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.612110] nix-grpc-daemon[771]: ts=2026-09-23T04:11:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.682524] nix-grpc-daemon[766]: ts=2026-09-23T04:11:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.687736] nix-grpc-daemon[766]: ts=2026-09-23T04:11:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.813082] nix-grpc-daemon[771]: ts=2026-09-23T04:11:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.818145] nix-grpc-daemon[771]: ts=2026-09-23T04:11:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.888420] nix-grpc-daemon[766]: ts=2026-09-23T04:11:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.893684] nix-grpc-daemon[766]: ts=2026-09-23T04:11:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.019121] nix-grpc-daemon[771]: ts=2026-09-23T04:11:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.024385] nix-grpc-daemon[771]: ts=2026-09-23T04:11:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.094511] nix-grpc-daemon[766]: ts=2026-09-23T04:11:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.099652] nix-grpc-daemon[766]: ts=2026-09-23T04:11:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.225251] nix-grpc-daemon[771]: ts=2026-09-23T04:11:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.230328] nix-grpc-daemon[771]: ts=2026-09-23T04:11:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.301927] nix-grpc-daemon[766]: ts=2026-09-23T04:11:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.306006] nix-grpc-daemon[766]: ts=2026-09-23T04:11:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 25.433190] nix-grpc-daemon[771]: ts=2026-09-23T04:11:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.437352] nix-grpc-daemon[771]: ts=2026-09-23T04:11:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.502192] nix-grpc-daemon[766]: ts=2026-09-23T04:11:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.507249] nix-grpc-daemon[766]: ts=2026-09-23T04:11:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.633675] nix-grpc-daemon[771]: ts=2026-09-23T04:11:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.638672] nix-grpc-daemon[771]: ts=2026-09-23T04:11:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.708107] nix-grpc-daemon[766]: ts=2026-09-23T04:11:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.713207] nix-grpc-daemon[766]: ts=2026-09-23T04:11:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.839567] nix-grpc-daemon[771]: ts=2026-09-23T04:11:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.844743] nix-grpc-daemon[771]: ts=2026-09-23T04:11:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.913869] nix-grpc-daemon[766]: ts=2026-09-23T04:11:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.919048] nix-grpc-daemon[766]: ts=2026-09-23T04:11:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.045637] nix-grpc-daemon[771]: ts=2026-09-23T04:11:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.050644] nix-grpc-daemon[771]: ts=2026-09-23T04:11:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.119664] nix-grpc-daemon[766]: ts=2026-09-23T04:11:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.124849] nix-grpc-daemon[766]: ts=2026-09-23T04:11:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.252990] nix-grpc-daemon[771]: ts=2026-09-23T04:11:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.257749] nix-grpc-daemon[771]: ts=2026-09-23T04:11:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.327332] nix-grpc-daemon[766]: ts=2026-09-23T04:11:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.331528] nix-grpc-daemon[766]: ts=2026-09-23T04:11:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 26.460464] nix-grpc-daemon[771]: ts=2026-09-23T04:11:41Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.464503] nix-grpc-daemon[771]: ts=2026-09-23T04:11:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 26.527578] nix-grpc-daemon[766]: ts=2026-09-23T04:11:41Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.532822] nix-grpc-daemon[766]: ts=2026-09-23T04:11:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.661225] nix-grpc-daemon[771]: ts=2026-09-23T04:11:41Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.666243] nix-grpc-daemon[771]: ts=2026-09-23T04:11:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.733627] nix-grpc-daemon[766]: ts=2026-09-23T04:11:41Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.738681] nix-grpc-daemon[766]: ts=2026-09-23T04:11:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.867046] nix-grpc-daemon[771]: ts=2026-09-23T04:11:41Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.872265] nix-grpc-daemon[771]: ts=2026-09-23T04:11:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.939376] nix-grpc-daemon[766]: ts=2026-09-23T04:11:41Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.944519] nix-grpc-daemon[766]: ts=2026-09-23T04:11:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.073146] nix-grpc-daemon[771]: ts=2026-09-23T04:11:41Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.078147] nix-grpc-daemon[771]: ts=2026-09-23T04:11:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.145126] nix-grpc-daemon[766]: ts=2026-09-23T04:11:41Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.150131] nix-grpc-daemon[766]: ts=2026-09-23T04:11:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.279014] nix-grpc-daemon[771]: ts=2026-09-23T04:11:41Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.284135] nix-grpc-daemon[771]: ts=2026-09-23T04:11:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.352557] nix-grpc-daemon[766]: ts=2026-09-23T04:11:41Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.356591] nix-grpc-daemon[766]: ts=2026-09-23T04:11:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 27.486649] nix-grpc-daemon[771]: ts=2026-09-23T04:11:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.490794] nix-grpc-daemon[771]: ts=2026-09-23T04:11:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 27.552984] nix-grpc-daemon[766]: ts=2026-09-23T04:11:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.558088] nix-grpc-daemon[766]: ts=2026-09-23T04:11:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.687221] nix-grpc-daemon[771]: ts=2026-09-23T04:11:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.695005] nix-grpc-daemon[771]: ts=2026-09-23T04:11:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.758877] nix-grpc-daemon[766]: ts=2026-09-23T04:11:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.764075] nix-grpc-daemon[766]: ts=2026-09-23T04:11:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.892811] nix-grpc-daemon[771]: ts=2026-09-23T04:11:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.897839] nix-grpc-daemon[771]: ts=2026-09-23T04:11:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.964717] nix-grpc-daemon[766]: ts=2026-09-23T04:11:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.969753] nix-grpc-daemon[766]: ts=2026-09-23T04:11:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.098759] nix-grpc-daemon[771]: ts=2026-09-23T04:11:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.103863] nix-grpc-daemon[771]: ts=2026-09-23T04:11:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.170351] nix-grpc-daemon[766]: ts=2026-09-23T04:11:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.175379] nix-grpc-daemon[766]: ts=2026-09-23T04:11:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.304772] nix-grpc-daemon[771]: ts=2026-09-23T04:11:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.309721] nix-grpc-daemon[771]: ts=2026-09-23T04:11:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.377913] nix-grpc-daemon[766]: ts=2026-09-23T04:11:42Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.382021] nix-grpc-daemon[766]: ts=2026-09-23T04:11:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.512425] nix-grpc-daemon[771]: ts=2026-09-23T04:11:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.516707] nix-grpc-daemon[771]: ts=2026-09-23T04:11:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 28.578195] nix-grpc-daemon[766]: ts=2026-09-23T04:11:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.583219] nix-grpc-daemon[766]: ts=2026-09-23T04:11:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) worker2 # [ 28.713101] nix-grpc-daemon[771]: ts=2026-09-23T04:11:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.718132] nix-grpc-daemon[771]: ts=2026-09-23T04:11:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.783929] nix-grpc-daemon[766]: ts=2026-09-23T04:11:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.789007] nix-grpc-daemon[766]: ts=2026-09-23T04:11:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.919037] nix-grpc-daemon[771]: ts=2026-09-23T04:11:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.924170] nix-grpc-daemon[771]: ts=2026-09-23T04:11:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.989669] nix-grpc-daemon[766]: ts=2026-09-23T04:11:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.994718] nix-grpc-daemon[766]: ts=2026-09-23T04:11:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.125018] nix-grpc-daemon[771]: ts=2026-09-23T04:11:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.130010] nix-grpc-daemon[771]: ts=2026-09-23T04:11:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.195301] nix-grpc-daemon[766]: ts=2026-09-23T04:11:43Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.200744] nix-grpc-daemon[766]: ts=2026-09-23T04:11:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.330818] nix-grpc-daemon[771]: ts=2026-09-23T04:11:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.335879] nix-grpc-daemon[771]: ts=2026-09-23T04:11:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.403149] nix-grpc-daemon[766]: ts=2026-09-23T04:11:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.407481] nix-grpc-daemon[766]: ts=2026-09-23T04:11:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.538537] nix-grpc-daemon[771]: ts=2026-09-23T04:11:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.542767] nix-grpc-daemon[771]: ts=2026-09-23T04:11:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.603587] nix-grpc-daemon[766]: ts=2026-09-23T04:11:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.608956] nix-grpc-daemon[766]: ts=2026-09-23T04:11:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 29.739076] nix-grpc-daemon[771]: ts=2026-09-23T04:11:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.744220] nix-grpc-daemon[771]: ts=2026-09-23T04:11:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.809670] nix-grpc-daemon[766]: ts=2026-09-23T04:11:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.814875] nix-grpc-daemon[766]: ts=2026-09-23T04:11:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.945175] nix-grpc-daemon[771]: ts=2026-09-23T04:11:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.950241] nix-grpc-daemon[771]: ts=2026-09-23T04:11:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.015591] nix-grpc-daemon[766]: ts=2026-09-23T04:11:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.020969] nix-grpc-daemon[766]: ts=2026-09-23T04:11:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.151165] nix-grpc-daemon[771]: ts=2026-09-23T04:11:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.156269] nix-grpc-daemon[771]: ts=2026-09-23T04:11:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.221547] nix-grpc-daemon[766]: ts=2026-09-23T04:11:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.226764] nix-grpc-daemon[766]: ts=2026-09-23T04:11:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.357101] nix-grpc-daemon[771]: ts=2026-09-23T04:11:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.362108] nix-grpc-daemon[771]: ts=2026-09-23T04:11:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.428843] nix-grpc-daemon[766]: ts=2026-09-23T04:11:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.433596] nix-grpc-daemon[766]: ts=2026-09-23T04:11:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.564677] nix-grpc-daemon[771]: ts=2026-09-23T04:11:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.568710] nix-grpc-daemon[771]: ts=2026-09-23T04:11:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.634177] nix-grpc-daemon[766]: ts=2026-09-23T04:11:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.639192] nix-grpc-daemon[766]: ts=2026-09-23T04:11:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 30.765127] nix-grpc-daemon[771]: ts=2026-09-23T04:11:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.770292] nix-grpc-daemon[771]: ts=2026-09-23T04:11:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.839908] nix-grpc-daemon[766]: ts=2026-09-23T04:11:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.845057] nix-grpc-daemon[766]: ts=2026-09-23T04:11:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.972822] nix-grpc-daemon[771]: ts=2026-09-23T04:11:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.978348] nix-grpc-daemon[771]: ts=2026-09-23T04:11:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.045627] nix-grpc-daemon[766]: ts=2026-09-23T04:11:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.050850] nix-grpc-daemon[766]: ts=2026-09-23T04:11:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.179070] nix-grpc-daemon[771]: ts=2026-09-23T04:11:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.184601] nix-grpc-daemon[771]: ts=2026-09-23T04:11:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.251389] nix-grpc-daemon[766]: ts=2026-09-23T04:11:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.256501] nix-grpc-daemon[766]: ts=2026-09-23T04:11:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.385381] nix-grpc-daemon[771]: ts=2026-09-23T04:11:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.390662] nix-grpc-daemon[771]: ts=2026-09-23T04:11:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.458828] nix-grpc-daemon[766]: ts=2026-09-23T04:11:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.463002] nix-grpc-daemon[766]: ts=2026-09-23T04:11:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.593036] nix-grpc-daemon[771]: ts=2026-09-23T04:11:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.597800] nix-grpc-daemon[771]: ts=2026-09-23T04:11:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.659248] nix-grpc-daemon[766]: ts=2026-09-23T04:11:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.664392] nix-grpc-daemon[766]: ts=2026-09-23T04:11:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 31.798637] nix-grpc-daemon[771]: ts=2026-09-23T04:11:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.803744] nix-grpc-daemon[771]: ts=2026-09-23T04:11:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker1 # [ 31.865023] nix-grpc-daemon[766]: ts=2026-09-23T04:11:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.870341] nix-grpc-daemon[766]: ts=2026-09-23T04:11:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.004876] nix-grpc-daemon[771]: ts=2026-09-23T04:11:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.010330] nix-grpc-daemon[771]: ts=2026-09-23T04:11:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.071057] nix-grpc-daemon[766]: ts=2026-09-23T04:11:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.076378] nix-grpc-daemon[766]: ts=2026-09-23T04:11:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.211425] nix-grpc-daemon[771]: ts=2026-09-23T04:11:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.216552] nix-grpc-daemon[771]: ts=2026-09-23T04:11:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.277048] nix-grpc-daemon[766]: ts=2026-09-23T04:11:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.282100] nix-grpc-daemon[766]: ts=2026-09-23T04:11:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.417555] nix-grpc-daemon[771]: ts=2026-09-23T04:11:46Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.422653] nix-grpc-daemon[771]: ts=2026-09-23T04:11:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.437286] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=scheduler_take_over lb # [ 32.439320] niks3-server[962]: 2026/09/23 04:11:47 INFO lead: acquired remote=192.168.1.3:57068 worker1 # [ 32.484491] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.488546] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.625460] nix-grpc-daemon[771]: ts=2026-09-23T04:11:47Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.629535] nix-grpc-daemon[771]: ts=2026-09-23T04:11:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.684952] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.690006] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.826153] nix-grpc-daemon[771]: ts=2026-09-23T04:11:47Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.831254] nix-grpc-daemon[771]: ts=2026-09-23T04:11:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 32.890686] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.895813] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 33.032391] nix-grpc-daemon[771]: ts=2026-09-23T04:11:47Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.037410] nix-grpc-daemon[771]: ts=2026-09-23T04:11:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.096676] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.101802] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.238532] nix-grpc-daemon[771]: ts=2026-09-23T04:11:47Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.257799] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 33.262272] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:55038 worker1 # [ 33.267924] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 33.302428] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.307790] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:55038 worker1 # [ 33.312755] nix-grpc-daemon[766]: ts=2026-09-23T04:11:47Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open' worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.06 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 11.43 seconds) subtest: balancer auth: client cert, bearer token, nothing, unknown CN worker1 # [ 34.973175] nix-grpc-daemon[766]: ts=2026-09-23T04:11:49Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 34.983639] systemd[1]: Started Nix Daemon. worker1 # [ 35.169953] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 35.187883] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 88 ms (attempt 1/5) worker1 # [ 35.280313] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 138 ms (attempt 2/5) worker1 # [ 35.420295] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 22 ms (attempt 3/5) worker1 # [ 35.443974] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 756 ms (attempt 4/5) worker1 # [ 36.204612] nix-grpc-daemon[766]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 36.209215] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 36.246278] nix-grpc-daemon[771]: ts=2026-09-23T04:11:50Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 36.261964] systemd[1]: Started Nix Daemon. worker2 # [ 36.436419] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 36.453041] nix-grpc-daemon[771]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 26 ms (attempt 1/5) worker2 # [ 36.482505] nix-grpc-daemon[771]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 166 ms (attempt 2/5) worker2 # [ 36.651121] nix-grpc-daemon[771]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 170 ms (attempt 3/5) worker2 # [ 36.830297] nix-grpc-daemon[771]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 111 ms (attempt 4/5) worker2 # [ 36.943949] nix-grpc-daemon[771]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 36.947539] nix-grpc-daemon[771]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 36.960169] nix-grpc-daemon[771]: ts=2026-09-23T04:11:51Z level=debug event=path_info_miss path=pka8rsnbb5rl489pq588cv7y1qycbh45-nixos-system-client-test error="error: path '/nix/store/pka8rsnbb5rl489pq588cv7y1qycbh45-nixos-system-client-test' is required, but there is no substituter that can build it" worker2 # [ 36.982846] nix-grpc-daemon[771]: ts=2026-09-23T04:11:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 36.994277] nix-grpc-daemon[771]: ts=2026-09-23T04:11:51Z level=debug event=path_info_miss path=pka8rsnbb5rl489pq588cv7y1qycbh45-nixos-system-client-test error="error: path '/nix/store/pka8rsnbb5rl489pq588cv7y1qycbh45-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.08 seconds) client: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key client: (finished: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds) worker1 # [ 37.310978] nix-grpc-daemon[766]: ts=2026-09-23T04:11:51Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:55052 worker1 # [ 37.327569] nix-grpc-daemon[766]: ts=2026-09-23T04:11:51Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:55052 worker1 # [ 37.340254] nix-grpc-daemon[766]: ts=2026-09-23T04:11:51Z level=debug event=path_info_miss path=pka8rsnbb5rl489pq588cv7y1qycbh45-nixos-system-client-test error="error: path '/nix/store/pka8rsnbb5rl489pq588cv7y1qycbh45-nixos-system-client-test' is required, but there is no substituter that can build it" worker1 # [ 37.354696] nix-grpc-daemon[766]: ts=2026-09-23T04:11:51Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:55052 worker1 # [ 37.362808] nix-grpc-daemon[766]: ts=2026-09-23T04:11:51Z level=debug event=path_info_miss path=pka8rsnbb5rl489pq588cv7y1qycbh45-nixos-system-client-test error="error: path '/nix/store/pka8rsnbb5rl489pq588cv7y1qycbh45-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: rm -r /var/lib/nix-grpc-store client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.03 seconds) worker1 # [ 37.581918] nix-grpc-daemon[766]: ts=2026-09-23T04:11:52Z level=info event=denied method=QueryMissing cn=- role=none worker2 # [ 37.771452] nix-grpc-daemon[771]: ts=2026-09-23T04:11:52Z level=info event=denied method=QueryMissing cn=stranger role=none (finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.12 seconds) subtest: DAG build is scheduled across workers and lands in the cache client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 worker1 # [ 37.962544] nix-grpc-daemon[766]: ts=2026-09-23T04:11:52Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55052 client # don't know how to build these paths: client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker2 # [ 37.978733] nix-grpc-daemon[771]: ts=2026-09-23T04:11:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49370 worker1 # [ 38.047377] nix-grpc-daemon[766]: ts=2026-09-23T04:11:52Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=0 worker1 # [ 38.054191] nix-grpc-daemon[766]: ts=2026-09-23T04:11:52Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 38.109678] nix-grpc-daemon[771]: ts=2026-09-23T04:11:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 38.132890] nix-grpc-daemon[771]: ts=2026-09-23T04:11:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49370 client # copying 1 paths to 'lb:50051'... worker2 # [ 38.159665] nix-grpc-daemon[771]: ts=2026-09-23T04:11:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'... worker2 # [ 38.189346] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 38.230109] nix-grpc-daemon[771]: ts=2026-09-23T04:11:52Z level=info event=niks3_push_started pid=853 worker2 # [ 38.492394] nix-grpc-daemon[853]: time=2026-09-23T04:11:53.067Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 38.640304] nix-daemon[840]: accepted connection from pid 866, user nix-grpc-daemon (trusted) worker2 # [ 38.664343] nix-daemon[840]: reaped child process 869, status = succeeded lb # [ 38.669639] niks3-server[962]: 2026/09/23 04:11:53 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 38.707567] nix-grpc-daemon[853]: time=2026-09-23T04:11:53.284Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 38.713020] nix-grpc-daemon[853]: time=2026-09-23T04:11:53.284Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)" lb # [ 38.761035] niks3-server[962]: 2026/09/23 04:11:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign lb # [ 38.779885] niks3-server[962]: 2026/09/23 04:11:53 INFO Signed narinfos id=1 count=1 worker2 # [ 38.788427] nix-grpc-daemon[853]: time=2026-09-23T04:11:53.365Z level=INFO msg="Uploading 1 narinfos" lb # [ 38.808172] niks3-server[962]: 2026/09/23 04:11:53 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls lb # [ 38.813202] niks3-server[962]: 2026/09/23 04:11:53 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst lb # [ 38.825476] niks3-server[962]: 2026/09/23 04:11:53 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete lb # [ 38.839005] niks3-server[962]: 2026/09/23 04:11:53 INFO Completed upload id=1 worker2 # [ 38.842099] nix-grpc-daemon[853]: time=2026-09-23T04:11:53.421Z level=INFO msg="Upload complete. (352ms)" lb # [ 38.841981] niks3-server[962]: 2026/09/23 04:11:53 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo worker2 # [ 38.857727] nix-grpc-daemon[771]: ts=2026-09-23T04:11:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_in=504 worker2 # [ 38.872537] nix-grpc-daemon[771]: ts=2026-09-23T04:11:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 38.888806] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 38.895725] nix-daemon[840]: reaped child process 851, status = succeeded client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051... worker2 # [ 38.911583] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) client # farm-a> worker2: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-t1 lb # [ 39.888238] envoy[906]: conn peer=[2001:db8:1::1]:40740 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=407 ms=2060 lb # [ 39.889494] envoy[906]: conn peer=[2001:db8:1::1]:40754 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=65 lb # [ 39.889868] envoy[906]: conn peer=[2001:db8:1::1]:40762 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=17 lb # [ 39.890137] envoy[906]: conn peer=[2001:db8:1::1]:40778 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=17 worker2 # [ 39.951371] nix-daemon[840]: reaped child process 873, status = succeeded worker2 # [ 40.101519] nix-daemon[840]: accepted connection from pid 889, user nix-grpc-daemon (trusted) worker2 # [ 40.126011] nix-daemon[840]: reaped child process 892, status = succeeded lb # [ 40.134025] niks3-server[962]: 2026/09/23 04:11:54 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 40.152544] nix-grpc-daemon[853]: time=2026-09-23T04:11:54.730Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 40.154231] nix-grpc-daemon[853]: time=2026-09-23T04:11:54.730Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)" lb # [ 40.188249] niks3-server[962]: 2026/09/23 04:11:54 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker2 # [ 40.216465] nix-grpc-daemon[853]: time=2026-09-23T04:11:54.794Z level=INFO msg="Uploading 1 narinfos" lb # [ 40.205449] niks3-server[962]: 2026/09/23 04:11:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign lb # [ 40.219367] niks3-server[962]: 2026/09/23 04:11:54 INFO Signed narinfos id=2 count=1 lb # [ 40.227801] niks3-server[962]: 2026/09/23 04:11:54 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst lb # [ 40.245889] niks3-server[962]: 2026/09/23 04:11:54 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete lb # [ 40.259750] niks3-server[962]: 2026/09/23 04:11:54 INFO Completed upload id=2 worker2 # [ 40.262564] nix-grpc-daemon[853]: time=2026-09-23T04:11:54.841Z level=INFO msg="Upload complete. (335ms)" lb # [ 40.263721] niks3-server[962]: 2026/09/23 04:11:54 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo lb # [ 40.277972] niks3-server[962]: 2026/09/23 04:11:54 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls worker1 # [ 40.289530] nix-grpc-daemon[766]: ts=2026-09-23T04:11:54Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=0 worker2 # [ 40.277525] nix-grpc-daemon[771]: ts=2026-09-23T04:11:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=1 worker2 # [ 40.302620] nix-daemon[840]: reaped child process 871, status = succeeded worker1 # [ 40.296187] nix-grpc-daemon[766]: ts=2026-09-23T04:11:54Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 40.340165] nix-grpc-daemon[771]: ts=2026-09-23T04:11:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 40.363175] nix-grpc-daemon[771]: ts=2026-09-23T04:11:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 40.380124] nix-grpc-daemon[853]: time=2026-09-23T04:11:54.959Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 40.393753] nix-grpc-daemon[771]: ts=2026-09-23T04:11:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'... worker2 # [ 40.424200] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 40.623316] nix-daemon[840]: accepted connection from pid 896, user nix-grpc-daemon (trusted) worker2 # [ 40.645993] nix-daemon[840]: reaped child process 899, status = succeeded lb # [ 40.652193] niks3-server[962]: 2026/09/23 04:11:55 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 40.670996] nix-grpc-daemon[853]: time=2026-09-23T04:11:55.249Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 40.673493] nix-grpc-daemon[853]: time=2026-09-23T04:11:55.249Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)" lb # [ 40.709746] niks3-server[962]: 2026/09/23 04:11:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign worker2 # [ 40.718995] nix-grpc-daemon[853]: time=2026-09-23T04:11:55.297Z level=INFO msg="Uploading 1 narinfos" lb # [ 40.723044] niks3-server[962]: 2026/09/23 04:11:55 INFO Signed narinfos id=3 count=1 lb # [ 40.729857] niks3-server[962]: 2026/09/23 04:11:55 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst lb # [ 40.737459] niks3-server[962]: 2026/09/23 04:11:55 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls lb # [ 40.748967] niks3-server[962]: 2026/09/23 04:11:55 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete lb # [ 40.764107] niks3-server[962]: 2026/09/23 04:11:55 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo lb # [ 40.771569] niks3-server[962]: 2026/09/23 04:11:55 INFO Completed upload id=3 worker2 # [ 40.774530] nix-grpc-daemon[853]: time=2026-09-23T04:11:55.353Z level=INFO msg="Upload complete. (321ms)" worker2 # [ 40.787797] nix-grpc-daemon[771]: ts=2026-09-23T04:11:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_in=624 worker2 # [ 40.803025] nix-grpc-daemon[771]: ts=2026-09-23T04:11:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 40.820839] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 40.838330] nix-daemon[840]: reaped child process 894, status = succeeded client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051... worker2 # [ 40.845018] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) client # farm-b> worker2: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-t1 worker2 # [ 42.104465] nix-daemon[840]: reaped child process 903, status = succeeded worker2 # [ 42.222073] nix-daemon[840]: accepted connection from pid 920, user nix-grpc-daemon (trusted) worker2 # [ 42.241767] nix-daemon[840]: reaped child process 923, status = succeeded lb # [ 42.238928] niks3-server[962]: 2026/09/23 04:11:56 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 42.251748] nix-grpc-daemon[853]: time=2026-09-23T04:11:56.830Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 42.253476] nix-grpc-daemon[853]: time=2026-09-23T04:11:56.830Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)" lb # [ 42.273990] niks3-server[962]: 2026/09/23 04:11:56 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv lb # [ 42.290440] niks3-server[962]: 2026/09/23 04:11:56 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls worker2 # [ 42.302427] nix-grpc-daemon[853]: time=2026-09-23T04:11:56.881Z level=INFO msg="Uploading 1 narinfos" lb # [ 42.302946] niks3-server[962]: 2026/09/23 04:11:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign lb # [ 42.310694] niks3-server[962]: 2026/09/23 04:11:56 INFO Signed narinfos id=4 count=1 lb # [ 42.315855] niks3-server[962]: 2026/09/23 04:11:56 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst lb # [ 42.324576] niks3-server[962]: 2026/09/23 04:11:56 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete lb # [ 42.330155] niks3-server[962]: 2026/09/23 04:11:56 INFO Completed upload id=4 lb # [ 42.332720] niks3-server[962]: 2026/09/23 04:11:56 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo worker2 # [ 42.333134] nix-grpc-daemon[853]: time=2026-09-23T04:11:56.912Z level=INFO msg="Upload complete. (275ms)" worker2 # [ 42.345343] nix-grpc-daemon[771]: ts=2026-09-23T04:11:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=1 worker1 # [ 42.347292] nix-grpc-daemon[766]: ts=2026-09-23T04:11:56Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=0 worker1 # [ 42.353544] nix-grpc-daemon[766]: ts=2026-09-23T04:11:56Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 42.365706] nix-daemon[840]: reaped child process 901, status = succeeded worker2 # [ 42.395019] nix-grpc-daemon[771]: ts=2026-09-23T04:11:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 42.410515] nix-grpc-daemon[771]: ts=2026-09-23T04:11:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 42.424857] nix-grpc-daemon[853]: time=2026-09-23T04:11:57.004Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'... worker2 # [ 42.436107] nix-grpc-daemon[771]: ts=2026-09-23T04:11:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 42.449740] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 42.601601] nix-daemon[840]: accepted connection from pid 927, user nix-grpc-daemon (trusted) worker2 # [ 42.621770] nix-daemon[840]: reaped child process 930, status = succeeded lb # [ 42.618335] niks3-server[962]: 2026/09/23 04:11:57 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 42.629898] nix-grpc-daemon[853]: time=2026-09-23T04:11:57.208Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 42.631379] nix-grpc-daemon[853]: time=2026-09-23T04:11:57.208Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)" lb # [ 42.646112] niks3-server[962]: 2026/09/23 04:11:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign worker2 # [ 42.655615] nix-grpc-daemon[853]: time=2026-09-23T04:11:57.234Z level=INFO msg="Uploading 1 narinfos" lb # [ 42.661006] niks3-server[962]: 2026/09/23 04:11:57 INFO Signed narinfos id=5 count=1 lb # [ 42.669110] niks3-server[962]: 2026/09/23 04:11:57 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls lb # [ 42.681186] niks3-server[962]: 2026/09/23 04:11:57 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst worker2 # [ 42.689359] nix-grpc-daemon[853]: time=2026-09-23T04:11:57.268Z level=INFO msg="Upload complete. (223ms)" lb # [ 42.687958] niks3-server[962]: 2026/09/23 04:11:57 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete lb # [ 42.695240] niks3-server[962]: 2026/09/23 04:11:57 INFO Completed upload id=5 worker2 # [ 42.698515] nix-grpc-daemon[771]: ts=2026-09-23T04:11:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 42.697956] niks3-server[962]: 2026/09/23 04:11:57 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo worker2 # [ 42.710580] nix-grpc-daemon[771]: ts=2026-09-23T04:11:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 42.719744] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 42.722601] nix-daemon[840]: reaped child process 925, status = succeeded client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051... client # farm-top> worker2: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker2 # [ 42.736194] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1 worker2 # [ 43.982119] nix-daemon[840]: reaped child process 934, status = succeeded worker2 # [ 44.095972] nix-daemon[840]: accepted connection from pid 950, user nix-grpc-daemon (trusted) worker2 # [ 44.113900] nix-daemon[840]: reaped child process 953, status = succeeded lb # [ 44.121323] niks3-server[962]: 2026/09/23 04:11:58 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 44.136419] nix-grpc-daemon[853]: time=2026-09-23T04:11:58.714Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 44.138078] nix-grpc-daemon[853]: time=2026-09-23T04:11:58.714Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)" lb # [ 44.157438] niks3-server[962]: 2026/09/23 04:11:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign worker2 # [ 44.166924] nix-grpc-daemon[853]: time=2026-09-23T04:11:58.745Z level=INFO msg="Uploading 1 narinfos" lb # [ 44.166745] niks3-server[962]: 2026/09/23 04:11:58 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst lb # [ 44.176120] niks3-server[962]: 2026/09/23 04:11:58 INFO Signed narinfos id=6 count=1 lb # [ 44.180483] niks3-server[962]: 2026/09/23 04:11:58 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls lb # [ 44.183077] niks3-server[962]: 2026/09/23 04:11:58 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker2 # [ 44.197147] nix-grpc-daemon[853]: time=2026-09-23T04:11:58.776Z level=INFO msg="Upload complete. (251ms)" lb # [ 44.196255] niks3-server[962]: 2026/09/23 04:11:58 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete lb # [ 44.198774] niks3-server[962]: 2026/09/23 04:11:58 INFO Completed upload id=6 lb # [ 44.202881] niks3-server[962]: 2026/09/23 04:11:58 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo worker2 # [ 44.209047] nix-grpc-daemon[771]: ts=2026-09-23T04:11:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=1 worker2 # [ 44.222580] nix-daemon[840]: reaped child process 932, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 6.48 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.20 seconds) client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.03 seconds) client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client # copying 3 paths... client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'... client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'... client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.25 seconds) (finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.05 seconds) subtest: repeat is answered Cached by the scheduler without building worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker1 # finding garbage collector roots... worker1 # deleting specified paths... worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker1 # deleting unused links... worker1 # note: hard linking is currently saving 0.0 KiB worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.24 seconds) worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker2 # finding garbage collector roots... worker2 # deleting specified paths... worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker2 # deleting unused links... worker2 # note: hard linking is currently saving 0.0 KiB worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.22 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 worker1 # [ 45.527283] nix-grpc-daemon[766]: ts=2026-09-23T04:12:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55052 client # don't know how to build these paths: client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker2 # [ 45.545151] nix-grpc-daemon[771]: ts=2026-09-23T04:12:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 45.553639] nix-grpc-daemon[853]: time=2026-09-23T04:12:00.130Z level=INFO msg="All 2 paths already cached" worker1 # [ 45.560826] nix-grpc-daemon[766]: ts=2026-09-23T04:12:00Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=1 client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 0.30 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.16 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: repeat is answered Cached by the scheduler without building, in 1.19 seconds) subtest: inputs already in the cache are not sent to the scheduler lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2 worker1 # [ 46.398825] nix-grpc-daemon[766]: ts=2026-09-23T04:12:01Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55052 client # don't know how to build these paths: client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker1 # [ 46.408547] nix-grpc-daemon[766]: ts=2026-09-23T04:12:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 46.435895] nix-grpc-daemon[766]: ts=2026-09-23T04:12:01Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=0 worker1 # [ 46.441707] nix-grpc-daemon[766]: ts=2026-09-23T04:12:01Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 46.487681] nix-grpc-daemon[771]: ts=2026-09-23T04:12:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 46.506294] nix-grpc-daemon[771]: ts=2026-09-23T04:12:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 46.521442] nix-grpc-daemon[853]: time=2026-09-23T04:12:01.100Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'... worker2 # [ 46.533306] nix-grpc-daemon[771]: ts=2026-09-23T04:12:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 46.550907] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 46.718815] nix-daemon[840]: accepted connection from pid 971, user nix-grpc-daemon (trusted) worker2 # [ 46.737899] nix-daemon[840]: reaped child process 974, status = succeeded lb # [ 46.742996] niks3-server[962]: 2026/09/23 04:12:01 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 46.759197] nix-grpc-daemon[853]: time=2026-09-23T04:12:01.337Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 46.760912] nix-grpc-daemon[853]: time=2026-09-23T04:12:01.337Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)" lb # [ 46.773116] niks3-server[962]: 2026/09/23 04:12:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign worker2 # [ 46.784883] nix-grpc-daemon[853]: time=2026-09-23T04:12:01.363Z level=INFO msg="Uploading 1 narinfos" lb # [ 46.785921] niks3-server[962]: 2026/09/23 04:12:01 INFO Signed narinfos id=7 count=1 lb # [ 46.796264] niks3-server[962]: 2026/09/23 04:12:01 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst lb # [ 46.809276] niks3-server[962]: 2026/09/23 04:12:01 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls lb # [ 46.821068] niks3-server[962]: 2026/09/23 04:12:01 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete worker2 # [ 46.828423] nix-grpc-daemon[853]: time=2026-09-23T04:12:01.407Z level=INFO msg="Upload complete. (258ms)" lb # [ 46.830430] niks3-server[962]: 2026/09/23 04:12:01 INFO Completed upload id=7 lb # [ 46.835584] niks3-server[962]: 2026/09/23 04:12:01 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo worker2 # [ 46.839630] nix-grpc-daemon[771]: ts=2026-09-23T04:12:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_in=752 worker2 # [ 46.855431] nix-grpc-daemon[771]: ts=2026-09-23T04:12:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 46.864311] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 46.866476] nix-daemon[840]: reaped child process 969, status = succeeded worker2 # [ 46.877560] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051... client # farm-top> worker2: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1b worker2 # [ 47.916926] nix-daemon[840]: reaped child process 978, status = succeeded worker2 # [ 48.033441] nix-daemon[840]: accepted connection from pid 994, user nix-grpc-daemon (trusted) worker2 # [ 48.051472] nix-daemon[840]: reaped child process 997, status = succeeded lb # [ 48.055227] niks3-server[962]: 2026/09/23 04:12:02 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 48.070272] nix-grpc-daemon[853]: time=2026-09-23T04:12:02.646Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 48.071865] nix-grpc-daemon[853]: time=2026-09-23T04:12:02.647Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)" lb # [ 48.096285] niks3-server[962]: 2026/09/23 04:12:02 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls worker2 # [ 48.111523] nix-grpc-daemon[853]: time=2026-09-23T04:12:02.690Z level=INFO msg="Uploading 1 narinfos" lb # [ 48.109577] niks3-server[962]: 2026/09/23 04:12:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign lb # [ 48.123303] niks3-server[962]: 2026/09/23 04:12:02 INFO Signed narinfos id=8 count=1 lb # [ 48.131490] niks3-server[962]: 2026/09/23 04:12:02 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv lb # [ 48.144714] niks3-server[962]: 2026/09/23 04:12:02 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst worker2 # [ 48.147612] nix-grpc-daemon[853]: time=2026-09-23T04:12:02.726Z level=INFO msg="Upload complete. (258ms)" lb # [ 48.149134] niks3-server[962]: 2026/09/23 04:12:02 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete lb # [ 48.154951] niks3-server[962]: 2026/09/23 04:12:02 INFO Completed upload id=8 lb # [ 48.158452] niks3-server[962]: 2026/09/23 04:12:02 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo worker2 # [ 48.161111] nix-grpc-daemon[771]: ts=2026-09-23T04:12:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=1 worker2 # [ 48.178891] nix-daemon[840]: reaped child process 976, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.00 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.17 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds) (finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.51 seconds) subtest: outputs read back through any worker carry the cache signature client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2 client # don't know how to build these paths: client # /nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker2 # [ 48.749881] nix-grpc-daemon[771]: ts=2026-09-23T04:12:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:49370 worker1 # [ 48.753257] nix-grpc-daemon[766]: ts=2026-09-23T04:12:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 48.776229] nix-grpc-daemon[766]: ts=2026-09-23T04:12:03Z level=debug event=want client=4 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv cached=0 worker1 # [ 48.786319] nix-grpc-daemon[766]: ts=2026-09-23T04:12:03Z level=debug event=assigned drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 48.794914] nix-grpc-daemon[766]: ts=2026-09-23T04:12:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 48.802716] nix-grpc-daemon[766]: ts=2026-09-23T04:12:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' to 'lb:50051'... worker1 # [ 48.814529] nix-grpc-daemon[766]: ts=2026-09-23T04:12:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 48.830502] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 48.855939] nix-grpc-daemon[766]: ts=2026-09-23T04:12:03Z level=info event=niks3_push_started pid=902 worker1 # [ 49.023212] nix-grpc-daemon[902]: time=2026-09-23T04:12:03.624Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 49.133222] nix-daemon[849]: accepted connection from pid 915, user nix-grpc-daemon (trusted) worker1 # [ 49.152635] nix-daemon[849]: reaped child process 918, status = succeeded lb # [ 49.159950] niks3-server[962]: 2026/09/23 04:12:03 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 49.169837] nix-grpc-daemon[902]: time=2026-09-23T04:12:03.771Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 49.171278] nix-grpc-daemon[902]: time=2026-09-23T04:12:03.771Z level=INFO msg="Uploading 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv (512B)" lb # [ 49.203384] niks3-server[962]: 2026/09/23 04:12:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign worker1 # [ 49.213705] nix-grpc-daemon[902]: time=2026-09-23T04:12:03.815Z level=INFO msg="Uploading 1 narinfos" lb # [ 49.217637] niks3-server[962]: 2026/09/23 04:12:03 INFO Signed narinfos id=9 count=1 lb # [ 49.224193] niks3-server[962]: 2026/09/23 04:12:03 INFO Registered completed upload object_key=nar/13mhmmy2qay13imbbdi287vhryz1hwnyg0w6agd9s0ph37wnsqll.nar.zst worker1 # [ 49.234290] nix-grpc-daemon[902]: time=2026-09-23T04:12:03.836Z level=INFO msg="Upload complete. (211ms)" lb # [ 49.236607] niks3-server[962]: 2026/09/23 04:12:03 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.ls lb # [ 49.239341] niks3-server[962]: 2026/09/23 04:12:03 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete lb # [ 49.244721] niks3-server[962]: 2026/09/23 04:12:03 INFO Completed upload id=9 worker1 # [ 49.244206] nix-grpc-daemon[766]: ts=2026-09-23T04:12:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 49.247535] niks3-server[962]: 2026/09/23 04:12:03 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.narinfo worker1 # [ 49.254836] nix-grpc-daemon[766]: ts=2026-09-23T04:12:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 49.260292] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 49.264539] nix-daemon[849]: reaped child process 900, status = succeeded client # building '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' on lb:50051... worker1 # [ 49.275754] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) client # farm-a> node-a: building 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv lb # [ 49.883912] envoy[906]: conn peer=[2001:db8:1::1]:40790 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9536 tx=4444 ms=6272 lb # [ 49.888885] envoy[906]: conn peer=[2001:db8:1::1]:43026 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=481 ms=56 lb # [ 49.893776] envoy[906]: conn peer=[2001:db8:1::1]:43040 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4198 tx=1920 ms=1773 client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-sig worker1 # [ 50.552401] nix-daemon[849]: reaped child process 922, status = succeeded worker1 # [ 50.651494] nix-daemon[849]: accepted connection from pid 938, user nix-grpc-daemon (trusted) worker1 # [ 50.668602] nix-daemon[849]: reaped child process 941, status = succeeded lb # [ 50.676869] niks3-server[962]: 2026/09/23 04:12:05 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 50.687191] nix-grpc-daemon[902]: time=2026-09-23T04:12:05.287Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 50.689990] nix-grpc-daemon[902]: time=2026-09-23T04:12:05.287Z level=INFO msg="Uploading 2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a (128B)" lb # [ 50.720884] niks3-server[962]: 2026/09/23 04:12:05 INFO Registered completed upload object_key=log/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker1 # [ 50.729692] nix-grpc-daemon[902]: time=2026-09-23T04:12:05.331Z level=INFO msg="Uploading 1 narinfos" lb # [ 50.734757] niks3-server[962]: 2026/09/23 04:12:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign lb # [ 50.742546] niks3-server[962]: 2026/09/23 04:12:05 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst lb # [ 50.752535] niks3-server[962]: 2026/09/23 04:12:05 INFO Signed narinfos id=10 count=1 lb # [ 50.756591] niks3-server[962]: 2026/09/23 04:12:05 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.ls worker1 # [ 50.755309] nix-grpc-daemon[902]: time=2026-09-23T04:12:05.357Z level=INFO msg="Upload complete. (269ms)" lb # [ 50.760196] niks3-server[962]: 2026/09/23 04:12:05 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete lb # [ 50.765361] niks3-server[962]: 2026/09/23 04:12:05 INFO Completed upload id=10 lb # [ 50.767487] niks3-server[962]: 2026/09/23 04:12:05 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.narinfo worker1 # [ 50.769134] nix-grpc-daemon[766]: ts=2026-09-23T04:12:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=1 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv assign_id=5 outputs=1 worker1 # [ 50.790522] nix-grpc-daemon[766]: ts=2026-09-23T04:12:05Z level=debug event=want client=4 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv cached=0 worker1 # [ 50.795634] nix-grpc-daemon[766]: ts=2026-09-23T04:12:05Z level=debug event=assigned drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 50.800647] nix-daemon[849]: reaped child process 920, status = succeeded worker1 # [ 50.835575] nix-grpc-daemon[766]: ts=2026-09-23T04:12:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 50.850969] nix-grpc-daemon[766]: ts=2026-09-23T04:12:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 50.864401] nix-grpc-daemon[902]: time=2026-09-23T04:12:05.466Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' to 'lb:50051'... worker1 # [ 50.875117] nix-grpc-daemon[766]: ts=2026-09-23T04:12:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 50.889953] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 51.044378] nix-daemon[849]: accepted connection from pid 947, user nix-grpc-daemon (trusted) worker1 # [ 51.067109] nix-daemon[849]: reaped child process 950, status = succeeded lb # [ 51.066977] niks3-server[962]: 2026/09/23 04:12:05 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 51.074883] nix-grpc-daemon[902]: time=2026-09-23T04:12:05.676Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 51.076656] nix-grpc-daemon[902]: time=2026-09-23T04:12:05.677Z level=INFO msg="Uploading z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv (624B)" lb # [ 51.096731] niks3-server[962]: 2026/09/23 04:12:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign worker1 # [ 51.105382] nix-grpc-daemon[902]: time=2026-09-23T04:12:05.707Z level=INFO msg="Uploading 1 narinfos" lb # [ 51.111652] niks3-server[962]: 2026/09/23 04:12:05 INFO Signed narinfos id=11 count=1 lb # [ 51.119964] niks3-server[962]: 2026/09/23 04:12:05 INFO Registered completed upload object_key=nar/0cnhkgsbmxv5hj7zw3fa59qfxikw6251mlhz2gz1v6pgn8c8shwp.nar.zst lb # [ 51.133003] niks3-server[962]: 2026/09/23 04:12:05 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.ls worker1 # [ 51.139311] nix-grpc-daemon[902]: time=2026-09-23T04:12:05.741Z level=INFO msg="Upload complete. (232ms)" lb # [ 51.141078] niks3-server[962]: 2026/09/23 04:12:05 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete lb # [ 51.146443] niks3-server[962]: 2026/09/23 04:12:05 INFO Completed upload id=11 worker1 # [ 51.146558] nix-grpc-daemon[766]: ts=2026-09-23T04:12:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 51.151221] niks3-server[962]: 2026/09/23 04:12:05 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.narinfo worker1 # [ 51.162648] nix-grpc-daemon[766]: ts=2026-09-23T04:12:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 51.169719] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 51.174677] nix-daemon[849]: reaped child process 945, status = succeeded client # building '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' on lb:50051... worker1 # [ 51.185269] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) client # farm-b> node-a: building z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-sig worker1 # [ 52.456915] nix-daemon[849]: reaped child process 954, status = succeeded worker1 # [ 52.568104] nix-daemon[849]: accepted connection from pid 970, user nix-grpc-daemon (trusted) worker1 # [ 52.583664] nix-daemon[849]: reaped child process 973, status = succeeded lb # [ 52.593963] niks3-server[962]: 2026/09/23 04:12:07 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 52.601213] nix-grpc-daemon[902]: time=2026-09-23T04:12:07.202Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 52.602937] nix-grpc-daemon[902]: time=2026-09-23T04:12:07.202Z level=INFO msg="Uploading h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b (176B)" worker1 # [ 52.642537] nix-grpc-daemon[902]: time=2026-09-23T04:12:07.244Z level=INFO msg="Uploading 1 narinfos" lb # [ 52.632157] niks3-server[962]: 2026/09/23 04:12:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign lb # [ 52.653772] niks3-server[962]: 2026/09/23 04:12:07 INFO Signed narinfos id=12 count=1 lb # [ 52.661025] niks3-server[962]: 2026/09/23 04:12:07 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.ls lb # [ 52.674652] niks3-server[962]: 2026/09/23 04:12:07 INFO Registered completed upload object_key=nar/0f75gpfkjj0g9xi2113yvvpdw0l16spc2f1xik1ifnzfbrvx4pfk.nar.zst worker1 # [ 52.682313] nix-grpc-daemon[902]: time=2026-09-23T04:12:07.284Z level=INFO msg="Upload complete. (284ms)" lb # [ 52.688906] niks3-server[962]: 2026/09/23 04:12:07 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete lb # [ 52.695477] niks3-server[962]: 2026/09/23 04:12:07 INFO Completed upload id=12 lb # [ 52.701837] niks3-server[962]: 2026/09/23 04:12:07 INFO Registered completed upload object_key=log/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker1 # [ 52.698554] nix-grpc-daemon[766]: ts=2026-09-23T04:12:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=1 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv assign_id=6 outputs=1 lb # [ 52.707457] niks3-server[962]: 2026/09/23 04:12:07 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.narinfo worker1 # [ 52.719788] nix-grpc-daemon[766]: ts=2026-09-23T04:12:07Z level=debug event=want client=4 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv cached=0 worker1 # [ 52.725055] nix-grpc-daemon[766]: ts=2026-09-23T04:12:07Z level=debug event=assigned drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 52.730386] nix-daemon[849]: reaped child process 952, status = succeeded worker1 # [ 52.760641] nix-grpc-daemon[766]: ts=2026-09-23T04:12:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 52.772322] nix-grpc-daemon[766]: ts=2026-09-23T04:12:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 52.786618] nix-grpc-daemon[902]: time=2026-09-23T04:12:07.389Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' to 'lb:50051'... worker1 # [ 52.798009] nix-grpc-daemon[766]: ts=2026-09-23T04:12:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 52.814675] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 52.971285] nix-daemon[849]: accepted connection from pid 979, user nix-grpc-daemon (trusted) worker1 # [ 52.986163] nix-daemon[849]: reaped child process 982, status = succeeded lb # [ 52.996253] niks3-server[962]: 2026/09/23 04:12:07 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 53.006008] nix-grpc-daemon[902]: time=2026-09-23T04:12:07.608Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 53.007637] nix-grpc-daemon[902]: time=2026-09-23T04:12:07.608Z level=INFO msg="Uploading knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv (752B)" lb # [ 53.036214] niks3-server[962]: 2026/09/23 04:12:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign worker1 # [ 53.041566] nix-grpc-daemon[902]: time=2026-09-23T04:12:07.643Z level=INFO msg="Uploading 1 narinfos" lb # [ 53.045884] niks3-server[962]: 2026/09/23 04:12:07 INFO Signed narinfos id=13 count=1 lb # [ 53.051915] niks3-server[962]: 2026/09/23 04:12:07 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.ls lb # [ 53.060219] niks3-server[962]: 2026/09/23 04:12:07 INFO Registered completed upload object_key=nar/0lpagyrngav67x80d1hq0njjlkmqdkvqawraz8dsp36ag50v6iwf.nar.zst lb # [ 53.070152] niks3-server[962]: 2026/09/23 04:12:07 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete worker1 # [ 53.075664] nix-grpc-daemon[902]: time=2026-09-23T04:12:07.678Z level=INFO msg="Upload complete. (244ms)" lb # [ 53.078795] niks3-server[962]: 2026/09/23 04:12:07 INFO Completed upload id=13 lb # [ 53.082451] niks3-server[962]: 2026/09/23 04:12:07 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.narinfo worker1 # [ 53.085557] nix-grpc-daemon[766]: ts=2026-09-23T04:12:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 53.103550] nix-grpc-daemon[766]: ts=2026-09-23T04:12:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 53.110378] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 53.115454] nix-daemon[849]: reaped child process 977, status = succeeded client # building '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' on lb:50051... worker1 # [ 53.128159] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) client # farm-top> node-a: building knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-sig worker1 # [ 54.615717] nix-daemon[849]: reaped child process 986, status = succeeded worker1 # [ 54.682361] nix-daemon[849]: accepted connection from pid 1002, user nix-grpc-daemon (trusted) worker1 # [ 54.698152] nix-daemon[849]: reaped child process 1005, status = succeeded lb # [ 54.706835] niks3-server[962]: 2026/09/23 04:12:09 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 54.716486] nix-grpc-daemon[902]: time=2026-09-23T04:12:09.317Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 54.719128] nix-grpc-daemon[902]: time=2026-09-23T04:12:09.317Z level=INFO msg="Uploading 4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top (232B)" lb # [ 54.741864] niks3-server[962]: 2026/09/23 04:12:09 INFO Registered completed upload object_key=nar/07ar364rwqqniwi2nkdpqlgv7ws7702n5lqz703ackfmssrh40v8.nar.zst lb # [ 54.756615] niks3-server[962]: 2026/09/23 04:12:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign worker1 # [ 54.761794] nix-grpc-daemon[902]: time=2026-09-23T04:12:09.364Z level=INFO msg="Uploading 1 narinfos" lb # [ 54.767464] niks3-server[962]: 2026/09/23 04:12:09 INFO Signed narinfos id=14 count=1 lb # [ 54.773628] niks3-server[962]: 2026/09/23 04:12:09 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.ls lb # [ 54.780635] niks3-server[962]: 2026/09/23 04:12:09 INFO Registered completed upload object_key=log/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv lb # [ 54.789462] niks3-server[962]: 2026/09/23 04:12:09 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete lb # [ 54.795055] niks3-server[962]: 2026/09/23 04:12:09 INFO Completed upload id=14 worker1 # [ 54.793708] nix-grpc-daemon[902]: time=2026-09-23T04:12:09.396Z level=INFO msg="Upload complete. (252ms)" lb # [ 54.798349] niks3-server[962]: 2026/09/23 04:12:09 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.narinfo worker1 # [ 54.809004] nix-grpc-daemon[766]: ts=2026-09-23T04:12:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=1 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv assign_id=7 outputs=1 worker1 # [ 54.822337] nix-daemon[849]: reaped child process 984, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2, in 6.32 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath, in 0.18 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker1 # [ 55.159148] nix-grpc-daemon[766]: ts=2026-09-23T04:12:09Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:38870 worker1 # [ 55.166623] nix-grpc-daemon[766]: ts=2026-09-23T04:12:09Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:38870 client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.19 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker2 # [ 55.372628] nix-grpc-daemon[771]: ts=2026-09-23T04:12:09Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:36490 client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked): client # /nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a client # /nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b client # /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker2 # [ 55.420594] nix-grpc-daemon[771]: ts=2026-09-23T04:12:09Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:36490 worker2 # [ 55.429613] nix-grpc-daemon[771]: copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'http://lb:5751'... worker2 # [ 55.446848] nix-grpc-daemon[771]: copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'http://lb:5751'... worker2 # [ 55.461512] nix-grpc-daemon[771]: copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.28 seconds) client: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top client # finding garbage collector roots... client # removing stale temporary roots file "/nix/var/nix/temproots/863" client # removing stale temporary roots file "/nix/var/nix/temproots/897" client # removing stale temporary roots file "/nix/var/nix/temproots/941" client # removing stale temporary roots file "/nix/var/nix/temproots/905" client # removing stale temporary roots file "/nix/var/nix/temproots/926" client # removing stale temporary roots file "/nix/var/nix/temproots/885" client # removing stale temporary roots file "/nix/var/nix/temproots/846" client # removing stale temporary roots file "/nix/var/nix/temproots/918" client # deleting specified paths... client # deleting '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' client # deleting unused links... client # note: hard linking is currently saving 0.0 KiB worker1 # [ 55.841107] nix-grpc-daemon[766]: ts=2026-09-23T04:12:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 55.847826] nix-grpc-daemon[766]: ts=2026-09-23T04:12:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 55.858603] nix-grpc-daemon[902]: time=2026-09-23T04:12:10.460Z level=INFO msg="All 1 paths already cached" worker2 # [ 55.871189] nix-grpc-daemon[771]: ts=2026-09-23T04:12:10Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49370 worker1 # [ 55.871869] nix-grpc-daemon[766]: ts=2026-09-23T04:12:10Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:55052 client # copying 3 paths... worker1 # [ 55.885526] nix-grpc-daemon[766]: ts=2026-09-23T04:12:10Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:55052 client # copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'grpc://lb:50051'... worker2 # [ 55.918673] nix-grpc-daemon[771]: ts=2026-09-23T04:12:10Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 55.926703] nix-grpc-daemon[771]: ts=2026-09-23T04:12:10Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_out=232 worker1 # [ 55.930089] nix-grpc-daemon[766]: ts=2026-09-23T04:12:10Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 55.937438] nix-grpc-daemon[766]: ts=2026-09-23T04:12:10Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=0 paths=1 nar_bytes_out=176 worker2 # [ 55.946981] nix-grpc-daemon[771]: ts=2026-09-23T04:12:10Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 55.953807] nix-grpc-daemon[771]: ts=2026-09-23T04:12:10Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_out=128 client # copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'grpc://lb:50051'... client # copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'grpc://lb:50051'... client: (finished: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.55 seconds) (finished: subtest: outputs read back through any worker carry the cache signature, in 7.52 seconds) subtest: two clients wanting the same drv build it once client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup client # [ 56.106223] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup1.service; invocation ID: 3db2f523220d4e318f9f6513015ddee4 client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.07 seconds) client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup worker1 # [ 56.289157] nix-grpc-daemon[766]: ts=2026-09-23T04:12:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55052 client # [ 56.300981] nix[994]: don't know how to build these paths: client # [ 56.302626] nix[994]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker1 # [ 56.306751] nix-grpc-daemon[766]: ts=2026-09-23T04:12:10Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker1 # [ 56.313038] nix-grpc-daemon[766]: ts=2026-09-23T04:12:10Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 56.354986] nix-grpc-daemon[771]: ts=2026-09-23T04:12:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 56.370370] nix-grpc-daemon[771]: ts=2026-09-23T04:12:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49370 client # [ 56.384294] nix[994]: copying 1 paths to 'lb:50051'... worker2 # [ 56.386164] nix-grpc-daemon[771]: ts=2026-09-23T04:12:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 client # [ 56.389433] nix[994]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'... worker2 # [ 56.401673] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 56.564426] nix-daemon[840]: accepted connection from pid 1004, user nix-grpc-daemon (trusted) worker2 # [ 56.584700] nix-daemon[840]: reaped child process 1007, status = succeeded lb # [ 56.588646] niks3-server[962]: 2026/09/23 04:12:11 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 56.600416] nix-grpc-daemon[853]: time=2026-09-23T04:12:11.179Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 56.601991] nix-grpc-daemon[853]: time=2026-09-23T04:12:11.179Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)" lb # [ 56.623612] niks3-server[962]: 2026/09/23 04:12:11 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst worker2 # [ 56.639077] nix-grpc-daemon[853]: time=2026-09-23T04:12:11.218Z level=INFO msg="Uploading 1 narinfos" lb # [ 56.638296] niks3-server[962]: 2026/09/23 04:12:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign lb # [ 56.649928] niks3-server[962]: 2026/09/23 04:12:11 INFO Signed narinfos id=15 count=1 lb # [ 56.658145] niks3-server[962]: 2026/09/23 04:12:11 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls worker2 # [ 56.666496] nix-grpc-daemon[853]: time=2026-09-23T04:12:11.245Z level=INFO msg="Upload complete. (247ms)" lb # [ 56.665378] niks3-server[962]: 2026/09/23 04:12:11 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete lb # [ 56.673689] niks3-server[962]: 2026/09/23 04:12:11 INFO Completed upload id=15 lb # [ 56.675684] niks3-server[962]: 2026/09/23 04:12:11 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo worker2 # [ 56.676581] nix-grpc-daemon[771]: ts=2026-09-23T04:12:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_in=432 worker2 # [ 56.692641] nix-grpc-daemon[771]: ts=2026-09-23T04:12:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 56.700400] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 56.708786] nix-daemon[840]: reaped child process 1002, status = succeeded worker2 # [ 56.717690] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) client # [ 56.717598] nix[994]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... client # [ 57.197725] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup2.service; invocation ID: cf90519e5b0c4d7087d757bd65aee4d9 client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.10 seconds) worker1 # [ 57.422189] nix-grpc-daemon[766]: ts=2026-09-23T04:12:12Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55052 client # [ 57.434489] nix[1010]: don't know how to build these paths: client # [ 57.435707] nix[1010]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker1 # [ 57.441702] nix-grpc-daemon[766]: ts=2026-09-23T04:12:12Z level=debug event=want client=6 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 client # [ 57.454694] nix[1010]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... worker2 # [ 57.453227] nix-grpc-daemon[771]: ts=2026-09-23T04:12:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 client: must succeed: sleep 2 lb # [ 59.886740] envoy[906]: conn peer=[2001:db8:1::1]:43052 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9571 tx=4368 ms=6073 lb # [ 59.891769] envoy[906]: conn peer=[2001:db8:1::1]:52350 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1051 tx=2889 ms=186 lb # [ 59.896647] envoy[906]: conn peer=[2001:db8:1::1]:52384 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=408 tx=329 ms=83 lb # [ 59.901334] envoy[906]: conn peer=[2001:db8:1::1]:52368 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=408 tx=375 ms=98 lb # [ 59.905996] envoy[906]: conn peer=[2001:db8:1::1]:52360 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=384 tx=409 ms=113 client: (finished: must succeed: sleep 2, in 2.05 seconds) client: waiting for success: ! systemctl is-active dedup1 dedup2 worker2 # [ 60.749266] nix-daemon[840]: reaped child process 1011, status = succeeded worker2 # [ 60.849679] nix-daemon[840]: accepted connection from pid 1041, user nix-grpc-daemon (trusted) worker2 # [ 60.865483] nix-daemon[840]: reaped child process 1044, status = succeeded lb # [ 60.870756] niks3-server[962]: 2026/09/23 04:12:15 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 60.884376] nix-grpc-daemon[853]: time=2026-09-23T04:12:15.462Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 60.885840] nix-grpc-daemon[853]: time=2026-09-23T04:12:15.462Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)" lb # [ 60.921930] niks3-server[962]: 2026/09/23 04:12:15 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker2 # [ 60.942340] nix-grpc-daemon[853]: time=2026-09-23T04:12:15.521Z level=INFO msg="Uploading 1 narinfos" lb # [ 60.931668] niks3-server[962]: 2026/09/23 04:12:15 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst lb # [ 60.945133] niks3-server[962]: 2026/09/23 04:12:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign lb # [ 60.951796] niks3-server[962]: 2026/09/23 04:12:15 INFO Signed narinfos id=16 count=1 lb # [ 60.956878] niks3-server[962]: 2026/09/23 04:12:15 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls lb # [ 60.967237] niks3-server[962]: 2026/09/23 04:12:15 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete lb # [ 60.972835] niks3-server[962]: 2026/09/23 04:12:15 INFO Completed upload id=16 worker2 # [ 60.974762] nix-grpc-daemon[853]: time=2026-09-23T04:12:15.553Z level=INFO msg="Upload complete. (242ms)" lb # [ 60.976234] niks3-server[962]: 2026/09/23 04:12:15 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo worker2 # [ 60.989103] nix-grpc-daemon[771]: ts=2026-09-23T04:12:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker2 # [ 61.005852] nix-grpc-daemon[771]: ts=2026-09-23T04:12:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker2 # [ 61.012880] nix-daemon[840]: reaped child process 1009, status = succeeded client # [ 61.023179] systemd[1]: dedup2.service: Deactivated successfully. client # [ 61.029122] systemd[1]: dedup2.service: Consumed 205ms CPU time over 3.826s wall clock time, 33.2M memory peak, 5.9K incoming IP traffic, 5.6K outgoing IP traffic. client # [ 61.067235] systemd[1]: dedup1.service: Deactivated successfully. client # [ 61.070369] systemd[1]: dedup1.service: Consumed 168ms CPU time over 4.963s wall clock time, 41.5M memory peak, 7.4K incoming IP traffic, 8.4K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.12 seconds) client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; } client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.06 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: two clients wanting the same drv build it once, in 6.11 seconds) subtest: --timeout and --max-silent-time reach the builder worker1 # [ 62.385771] nix-grpc-daemon[766]: ts=2026-09-23T04:12:16Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 62.402328] nix-grpc-daemon[766]: ts=2026-09-23T04:12:17Z level=debug event=want client=7 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv cached=0 worker1 # [ 62.409449] nix-grpc-daemon[766]: ts=2026-09-23T04:12:17Z level=debug event=assigned drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 62.447288] nix-grpc-daemon[766]: ts=2026-09-23T04:12:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 62.462596] nix-grpc-daemon[766]: ts=2026-09-23T04:12:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 62.477605] nix-grpc-daemon[766]: ts=2026-09-23T04:12:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 62.492452] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 62.654105] nix-daemon[849]: accepted connection from pid 1036, user nix-grpc-daemon (trusted) worker1 # [ 62.674226] nix-daemon[849]: reaped child process 1039, status = succeeded lb # [ 62.674416] niks3-server[962]: 2026/09/23 04:12:17 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 62.683173] nix-grpc-daemon[902]: time=2026-09-23T04:12:17.285Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 62.684792] nix-grpc-daemon[902]: time=2026-09-23T04:12:17.285Z level=INFO msg="Uploading sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv (424B)" lb # [ 62.706282] niks3-server[962]: 2026/09/23 04:12:17 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.ls worker1 # [ 62.715183] nix-grpc-daemon[902]: time=2026-09-23T04:12:17.317Z level=INFO msg="Uploading 1 narinfos" lb # [ 62.719834] niks3-server[962]: 2026/09/23 04:12:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign lb # [ 62.731285] niks3-server[962]: 2026/09/23 04:12:17 INFO Signed narinfos id=17 count=1 lb # [ 62.740386] niks3-server[962]: 2026/09/23 04:12:17 INFO Registered completed upload object_key=nar/1m12xmj6yzmb9y2j8ly0yvh5fyzy1nqk7ajhqwr4wn3b5h0bsch5.nar.zst worker1 # [ 62.746925] nix-grpc-daemon[902]: time=2026-09-23T04:12:17.349Z level=INFO msg="Upload complete. (237ms)" lb # [ 62.752584] niks3-server[962]: 2026/09/23 04:12:17 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete lb # [ 62.758877] niks3-server[962]: 2026/09/23 04:12:17 INFO Completed upload id=17 worker1 # [ 62.756465] nix-grpc-daemon[766]: ts=2026-09-23T04:12:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=0 paths=1 nar_bytes_in=424 lb # [ 62.763720] niks3-server[962]: 2026/09/23 04:12:17 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.narinfo worker1 # [ 62.774043] nix-grpc-daemon[766]: ts=2026-09-23T04:12:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 62.783406] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 62.787185] nix-daemon[849]: reaped child process 1034, status = succeeded worker1 # [ 62.799111] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 68.617060] nix-grpc-daemon[766]: ts=2026-09-23T04:12:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=5 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv assign_id=9 outputs=0 worker1 # [ 68.634302] nix-daemon[849]: reaped child process 1041, status = succeeded worker1 # [ 68.635448] nix-daemon[849]: reaped child process 1043, status = succeeded worker1 # [ 68.890527] nix-grpc-daemon[766]: ts=2026-09-23T04:12:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 68.907516] nix-grpc-daemon[766]: ts=2026-09-23T04:12:23Z level=debug event=want client=8 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv cached=0 worker2 # [ 68.917472] nix-grpc-daemon[771]: ts=2026-09-23T04:12:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker1 # [ 68.913981] nix-grpc-daemon[766]: ts=2026-09-23T04:12:23Z level=debug event=assigned drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 68.932718] nix-grpc-daemon[771]: ts=2026-09-23T04:12:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 68.946921] nix-grpc-daemon[771]: ts=2026-09-23T04:12:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 68.961811] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 69.126145] nix-daemon[840]: accepted connection from pid 1057, user nix-grpc-daemon (trusted) worker2 # [ 69.146439] nix-daemon[840]: reaped child process 1060, status = succeeded lb # [ 69.149872] niks3-server[962]: 2026/09/23 04:12:23 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 69.160390] nix-grpc-daemon[853]: time=2026-09-23T04:12:23.739Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 69.161947] nix-grpc-daemon[853]: time=2026-09-23T04:12:23.739Z level=INFO msg="Uploading pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv (424B)" lb # [ 69.178642] niks3-server[962]: 2026/09/23 04:12:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign worker2 # [ 69.190607] nix-grpc-daemon[853]: time=2026-09-23T04:12:23.769Z level=INFO msg="Uploading 1 narinfos" lb # [ 69.191408] niks3-server[962]: 2026/09/23 04:12:23 INFO Signed narinfos id=18 count=1 lb # [ 69.201157] niks3-server[962]: 2026/09/23 04:12:23 INFO Registered completed upload object_key=nar/14f5yrnllfav3svmw4v4svw15r4zlf5yv1p972jy14mmq6gdw943.nar.zst lb # [ 69.212797] niks3-server[962]: 2026/09/23 04:12:23 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.ls lb # [ 69.223293] niks3-server[962]: 2026/09/23 04:12:23 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete worker2 # [ 69.225531] nix-grpc-daemon[853]: time=2026-09-23T04:12:23.804Z level=INFO msg="Upload complete. (246ms)" lb # [ 69.232515] niks3-server[962]: 2026/09/23 04:12:23 INFO Completed upload id=18 lb # [ 69.235150] niks3-server[962]: 2026/09/23 04:12:23 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.narinfo worker2 # [ 69.235355] nix-grpc-daemon[771]: ts=2026-09-23T04:12:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 69.252968] nix-grpc-daemon[771]: ts=2026-09-23T04:12:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 69.262538] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 69.268451] nix-daemon[840]: reaped child process 1055, status = succeeded worker2 # [ 69.282893] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) lb # [ 69.883724] envoy[906]: conn peer=[2001:db8:1::1]:52396 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1220 ms=3580 lb # [ 69.888920] envoy[906]: conn peer=[2001:db8:1::1]:52392 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3044 tx=1565 ms=4754 lb # [ 69.893993] envoy[906]: conn peer=[2001:db8:1::1]:52528 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3027 tx=976 ms=6294 worker2 # [ 75.700667] nix-grpc-daemon[771]: ts=2026-09-23T04:12:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=6 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv assign_id=10 outputs=0 worker2 # [ 75.733916] nix-daemon[840]: reaped child process 1062, status = succeeded worker2 # [ 75.737013] nix-daemon[840]: reaped child process 1064, status = succeeded (finished: subtest: --timeout and --max-silent-time reach the builder, in 13.77 seconds) subtest: the first client leaving does not fail the second client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share client # [ 75.994943] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share1.service; invocation ID: d22e07145bbe411484c84dedfe503edb client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 0.08 seconds) client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share worker2 # [ 76.204860] nix-grpc-daemon[771]: ts=2026-09-23T04:12:30Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:49370 client # [ 76.212908] nix[1082]: don't know how to build these paths: client # [ 76.213942] nix[1082]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker1 # [ 76.225119] nix-grpc-daemon[766]: ts=2026-09-23T04:12:30Z level=debug event=want client=9 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 worker1 # [ 76.231601] nix-grpc-daemon[766]: ts=2026-09-23T04:12:30Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 76.238478] nix-grpc-daemon[766]: ts=2026-09-23T04:12:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 76.249384] nix-grpc-daemon[766]: ts=2026-09-23T04:12:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 client # [ 76.265515] nix[1082]: copying 1 paths to 'lb:50051'... client # [ 76.271799] nix[1082]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'... worker1 # [ 76.265096] nix-grpc-daemon[766]: ts=2026-09-23T04:12:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 76.279966] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 76.443244] nix-daemon[849]: accepted connection from pid 1070, user nix-grpc-daemon (trusted) worker1 # [ 76.461695] nix-daemon[849]: reaped child process 1073, status = succeeded lb # [ 76.469097] niks3-server[962]: 2026/09/23 04:12:31 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 76.479761] nix-grpc-daemon[902]: time=2026-09-23T04:12:31.081Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 76.481402] nix-grpc-daemon[902]: time=2026-09-23T04:12:31.081Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)" lb # [ 76.502201] niks3-server[962]: 2026/09/23 04:12:31 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst lb # [ 76.517973] niks3-server[962]: 2026/09/23 04:12:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign worker1 # [ 76.523273] nix-grpc-daemon[902]: time=2026-09-23T04:12:31.125Z level=INFO msg="Uploading 1 narinfos" lb # [ 76.529752] niks3-server[962]: 2026/09/23 04:12:31 INFO Signed narinfos id=19 count=1 lb # [ 76.538465] niks3-server[962]: 2026/09/23 04:12:31 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls lb # [ 76.550392] niks3-server[962]: 2026/09/23 04:12:31 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete lb # [ 76.560291] niks3-server[962]: 2026/09/23 04:12:31 INFO Completed upload id=19 worker1 # [ 76.558470] nix-grpc-daemon[902]: time=2026-09-23T04:12:31.160Z level=INFO msg="Upload complete. (259ms)" lb # [ 76.566405] niks3-server[962]: 2026/09/23 04:12:31 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo worker1 # [ 76.568396] nix-grpc-daemon[766]: ts=2026-09-23T04:12:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=0 paths=1 nar_bytes_in=432 worker1 # [ 76.587216] nix-grpc-daemon[766]: ts=2026-09-23T04:12:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 76.597357] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 76.600452] nix-daemon[849]: reaped child process 1068, status = succeeded client # [ 76.616151] nix[1082]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... worker1 # [ 76.614760] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) client # [ 77.093141] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share2.service; invocation ID: 01527c429db64ab8a320d2719cbff831 client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 1.11 seconds) worker2 # [ 77.312898] nix-grpc-daemon[771]: ts=2026-09-23T04:12:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:49370 client # [ 77.321037] nix[1098]: don't know how to build these paths: client # [ 77.322463] nix[1098]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker1 # [ 77.328579] nix-grpc-daemon[766]: ts=2026-09-23T04:12:31Z level=debug event=want client=10 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 client # [ 77.341391] nix[1098]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... worker1 # [ 77.336097] nix-grpc-daemon[766]: ts=2026-09-23T04:12:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 client: must succeed: sleep 2; systemctl kill -s INT share1 lb # [ 79.888202] envoy[906]: conn peer=[2001:db8:1::1]:52534 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2959 tx=937 ms=6825 client # [ 80.479399] systemd[1]: share1.service: Sent signal SIGINT to main process 1082 (nix) on client request. client # [ 80.485834] nix[1082]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.09 seconds) client: waiting for success: ! systemctl is-active share1 client # [ 80.499620] nix[1082]: error: interrupted by the user client # [ 80.564870] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE client # [ 80.568959] systemd[1]: share1.service: Failed with result 'exit-code'. client # [ 80.570808] systemd[1]: share1.service: Consumed 185ms CPU time over 4.573s wall clock time, 41.4M memory peak, 6.3K incoming IP traffic, 8.2K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share1, in 1.12 seconds) worker1 # [ 81.839087] nix-daemon[849]: reaped child process 1077, status = succeeded client: waiting for success: ! systemctl is-active share2 worker1 # [ 81.935050] nix-daemon[849]: accepted connection from pid 1113, user nix-grpc-daemon (trusted) worker1 # [ 81.953743] nix-daemon[849]: reaped child process 1116, status = succeeded lb # [ 81.961007] niks3-server[962]: 2026/09/23 04:12:36 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 81.974138] nix-grpc-daemon[902]: time=2026-09-23T04:12:36.574Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 81.975590] nix-grpc-daemon[902]: time=2026-09-23T04:12:36.574Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)" lb # [ 82.001595] niks3-server[962]: 2026/09/23 04:12:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign worker1 # [ 82.006995] nix-grpc-daemon[902]: time=2026-09-23T04:12:36.609Z level=INFO msg="Uploading 1 narinfos" lb # [ 82.011664] niks3-server[962]: 2026/09/23 04:12:36 INFO Signed narinfos id=20 count=1 lb # [ 82.018850] niks3-server[962]: 2026/09/23 04:12:36 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls lb # [ 82.027138] niks3-server[962]: 2026/09/23 04:12:36 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv lb # [ 82.034122] niks3-server[962]: 2026/09/23 04:12:36 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst lb # [ 82.041857] niks3-server[962]: 2026/09/23 04:12:36 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete worker1 # [ 82.047125] nix-grpc-daemon[902]: time=2026-09-23T04:12:36.649Z level=INFO msg="Upload complete. (214ms)" lb # [ 82.051825] niks3-server[962]: 2026/09/23 04:12:36 INFO Completed upload id=20 lb # [ 82.056404] niks3-server[962]: 2026/09/23 04:12:36 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo worker1 # [ 82.061033] nix-grpc-daemon[766]: ts=2026-09-23T04:12:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=5 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 worker1 # [ 82.072400] nix-grpc-daemon[766]: ts=2026-09-23T04:12:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=4 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 worker1 # [ 82.079301] nix-daemon[849]: reaped child process 1075, status = succeeded client # [ 82.138586] systemd[1]: share2.service: Deactivated successfully. client # [ 82.143509] systemd[1]: share2.service: Consumed 200ms CPU time over 5.049s wall clock time, 33.3M memory peak, 6K incoming IP traffic, 5.7K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share2, in 1.12 seconds) client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; } client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.05 seconds) (finished: subtest: the first client leaving does not fail the second, in 7.11 seconds) subtest: build hook: nix-daemon with builders = grpc://lb client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log client # [ 83.175088] systemd[1]: Started Nix Daemon. client # [ 83.292179] nix-daemon[1150]: accepted connection from pid 1147, user root (trusted) worker2 # [ 83.434738] nix-grpc-daemon[771]: ts=2026-09-23T04:12:38Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 83.442373] nix-grpc-daemon[771]: ts=2026-09-23T04:12:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49370 worker1 # [ 83.451201] nix-grpc-daemon[766]: ts=2026-09-23T04:12:38Z level=debug event=want client=11 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=0 worker1 # [ 83.457594] nix-grpc-daemon[766]: ts=2026-09-23T04:12:38Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 83.464875] nix-grpc-daemon[771]: ts=2026-09-23T04:12:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 83.480857] nix-grpc-daemon[771]: ts=2026-09-23T04:12:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 83.495715] nix-grpc-daemon[771]: ts=2026-09-23T04:12:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 83.510964] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 83.674725] nix-daemon[840]: accepted connection from pid 1103, user nix-grpc-daemon (trusted) worker2 # [ 83.695065] nix-daemon[840]: reaped child process 1106, status = succeeded lb # [ 83.701578] niks3-server[962]: 2026/09/23 04:12:38 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 83.715616] nix-grpc-daemon[853]: time=2026-09-23T04:12:38.293Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 83.717397] nix-grpc-daemon[853]: time=2026-09-23T04:12:38.293Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)" lb # [ 83.733240] niks3-server[962]: 2026/09/23 04:12:38 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst worker2 # [ 83.748845] nix-grpc-daemon[853]: time=2026-09-23T04:12:38.327Z level=INFO msg="Uploading 1 narinfos" lb # [ 83.749524] niks3-server[962]: 2026/09/23 04:12:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign lb # [ 83.759251] niks3-server[962]: 2026/09/23 04:12:38 INFO Signed narinfos id=21 count=1 lb # [ 83.764841] niks3-server[962]: 2026/09/23 04:12:38 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls worker2 # [ 83.776502] nix-grpc-daemon[853]: time=2026-09-23T04:12:38.355Z level=INFO msg="Upload complete. (246ms)" lb # [ 83.775204] niks3-server[962]: 2026/09/23 04:12:38 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete lb # [ 83.779816] niks3-server[962]: 2026/09/23 04:12:38 INFO Completed upload id=21 lb # [ 83.783785] niks3-server[962]: 2026/09/23 04:12:38 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo worker2 # [ 83.786453] nix-grpc-daemon[771]: ts=2026-09-23T04:12:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 83.801383] nix-grpc-daemon[771]: ts=2026-09-23T04:12:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 83.807456] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 83.812526] nix-daemon[840]: reaped child process 1101, status = succeeded worker2 # [ 83.823103] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 85.288368] nix-daemon[840]: reaped child process 1110, status = succeeded worker2 # [ 85.381833] nix-daemon[840]: accepted connection from pid 1126, user nix-grpc-daemon (trusted) worker2 # [ 85.401917] nix-daemon[840]: reaped child process 1129, status = succeeded lb # [ 85.398334] niks3-server[962]: 2026/09/23 04:12:40 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 85.411557] nix-grpc-daemon[853]: time=2026-09-23T04:12:39.989Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 85.415060] nix-grpc-daemon[853]: time=2026-09-23T04:12:39.990Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)" lb # [ 85.443768] niks3-server[962]: 2026/09/23 04:12:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign worker2 # [ 85.454024] nix-grpc-daemon[853]: time=2026-09-23T04:12:40.033Z level=INFO msg="Uploading 1 narinfos" lb # [ 85.459438] niks3-server[962]: 2026/09/23 04:12:40 INFO Signed narinfos id=22 count=1 lb # [ 85.466082] niks3-server[962]: 2026/09/23 04:12:40 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls lb # [ 85.472251] niks3-server[962]: 2026/09/23 04:12:40 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv lb # [ 85.476562] niks3-server[962]: 2026/09/23 04:12:40 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst lb # [ 85.483611] niks3-server[962]: 2026/09/23 04:12:40 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete worker2 # [ 85.490900] nix-grpc-daemon[853]: time=2026-09-23T04:12:40.070Z level=INFO msg="Upload complete. (254ms)" lb # [ 85.491357] niks3-server[962]: 2026/09/23 04:12:40 INFO Completed upload id=22 lb # [ 85.493499] niks3-server[962]: 2026/09/23 04:12:40 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo worker2 # [ 85.501083] nix-grpc-daemon[771]: ts=2026-09-23T04:12:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=12 outputs=1 worker1 # [ 85.506237] nix-grpc-daemon[766]: ts=2026-09-23T04:12:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:55052 worker2 # [ 85.519464] nix-daemon[840]: reaped child process 1108, status = succeeded worker1 # [ 85.519732] nix-grpc-daemon[766]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'... worker1 # [ 85.559728] nix-grpc-daemon[766]: ts=2026-09-23T04:12:40Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 85.566667] nix-grpc-daemon[766]: ts=2026-09-23T04:12:40Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=0 paths=1 nar_bytes_out=128 worker2 # [ 85.850007] nix-grpc-daemon[771]: ts=2026-09-23T04:12:40Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:49370 worker1 # [ 85.850622] nix-grpc-daemon[766]: ts=2026-09-23T04:12:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 85.861232] nix-grpc-daemon[902]: time=2026-09-23T04:12:40.462Z level=INFO msg="All 1 paths already cached" worker1 # [ 85.879007] nix-grpc-daemon[766]: ts=2026-09-23T04:12:40Z level=debug event=want client=12 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=0 worker1 # [ 85.885199] nix-grpc-daemon[766]: ts=2026-09-23T04:12:40Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 85.931302] nix-grpc-daemon[771]: ts=2026-09-23T04:12:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 85.946914] nix-grpc-daemon[771]: ts=2026-09-23T04:12:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 85.961312] nix-grpc-daemon[853]: time=2026-09-23T04:12:40.540Z level=INFO msg="All 1 paths already cached" worker2 # [ 85.970766] nix-grpc-daemon[771]: ts=2026-09-23T04:12:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 85.984785] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 86.147379] nix-daemon[840]: accepted connection from pid 1133, user nix-grpc-daemon (trusted) worker2 # [ 86.166685] nix-daemon[840]: reaped child process 1136, status = succeeded lb # [ 86.167285] niks3-server[962]: 2026/09/23 04:12:40 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 86.178829] nix-grpc-daemon[853]: time=2026-09-23T04:12:40.757Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 86.180432] nix-grpc-daemon[853]: time=2026-09-23T04:12:40.757Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)" lb # [ 86.197457] niks3-server[962]: 2026/09/23 04:12:40 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls worker2 # [ 86.211284] nix-grpc-daemon[853]: time=2026-09-23T04:12:40.790Z level=INFO msg="Uploading 1 narinfos" lb # [ 86.212289] niks3-server[962]: 2026/09/23 04:12:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign lb # [ 86.223460] niks3-server[962]: 2026/09/23 04:12:40 INFO Signed narinfos id=23 count=1 lb # [ 86.231094] niks3-server[962]: 2026/09/23 04:12:40 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst worker2 # [ 86.241127] nix-grpc-daemon[853]: time=2026-09-23T04:12:40.820Z level=INFO msg="Upload complete. (238ms)" lb # [ 86.240883] niks3-server[962]: 2026/09/23 04:12:40 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete lb # [ 86.248809] niks3-server[962]: 2026/09/23 04:12:40 INFO Completed upload id=23 lb # [ 86.251133] niks3-server[962]: 2026/09/23 04:12:40 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo worker2 # [ 86.251679] nix-grpc-daemon[771]: ts=2026-09-23T04:12:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_in=624 worker2 # [ 86.266851] nix-grpc-daemon[771]: ts=2026-09-23T04:12:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 86.275557] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 86.278958] nix-daemon[840]: reaped child process 1131, status = succeeded worker2 # [ 86.289482] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 87.763962] nix-daemon[840]: reaped child process 1140, status = succeeded worker2 # [ 87.869999] nix-daemon[840]: accepted connection from pid 1156, user nix-grpc-daemon (trusted) worker2 # [ 87.886867] nix-daemon[840]: reaped child process 1159, status = succeeded lb # [ 87.890449] niks3-server[962]: 2026/09/23 04:12:42 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 87.904426] nix-grpc-daemon[853]: time=2026-09-23T04:12:42.481Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 87.906139] nix-grpc-daemon[853]: time=2026-09-23T04:12:42.481Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)" worker2 # [ 87.942581] nix-grpc-daemon[853]: time=2026-09-23T04:12:42.521Z level=INFO msg="Uploading 1 narinfos" lb # [ 87.932501] niks3-server[962]: 2026/09/23 04:12:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign lb # [ 87.945057] niks3-server[962]: 2026/09/23 04:12:42 INFO Signed narinfos id=24 count=1 lb # [ 87.954304] niks3-server[962]: 2026/09/23 04:12:42 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst lb # [ 87.965863] niks3-server[962]: 2026/09/23 04:12:42 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv lb # [ 87.972182] niks3-server[962]: 2026/09/23 04:12:42 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls lb # [ 87.982800] niks3-server[962]: 2026/09/23 04:12:42 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete worker2 # [ 87.992392] nix-grpc-daemon[853]: time=2026-09-23T04:12:42.571Z level=INFO msg="Upload complete. (291ms)" lb # [ 87.990410] niks3-server[962]: 2026/09/23 04:12:42 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo lb # [ 87.994716] niks3-server[962]: 2026/09/23 04:12:42 INFO Completed upload id=24 worker2 # [ 88.005213] nix-grpc-daemon[771]: ts=2026-09-23T04:12:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=13 outputs=1 worker2 # [ 88.024814] nix-grpc-daemon[771]: ts=2026-09-23T04:12:42Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 88.030167] nix-daemon[840]: reaped child process 1138, status = succeeded worker2 # [ 88.041106] nix-grpc-daemon[771]: ts=2026-09-23T04:12:42Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 88.048240] nix-grpc-daemon[771]: ts=2026-09-23T04:12:42Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_out=176 worker1 # [ 88.320356] nix-grpc-daemon[766]: ts=2026-09-23T04:12:42Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 88.326802] nix-grpc-daemon[766]: ts=2026-09-23T04:12:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 88.343836] nix-grpc-daemon[902]: time=2026-09-23T04:12:42.945Z level=INFO msg="All 1 paths already cached" worker1 # [ 88.361980] nix-grpc-daemon[766]: ts=2026-09-23T04:12:42Z level=debug event=want client=13 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=0 worker2 # [ 88.369771] nix-grpc-daemon[771]: ts=2026-09-23T04:12:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker1 # [ 88.368425] nix-grpc-daemon[766]: ts=2026-09-23T04:12:42Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 88.385622] nix-grpc-daemon[771]: ts=2026-09-23T04:12:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 88.401903] nix-grpc-daemon[853]: time=2026-09-23T04:12:42.981Z level=INFO msg="All 2 paths already cached" worker2 # [ 88.412124] nix-grpc-daemon[771]: ts=2026-09-23T04:12:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 88.426209] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 88.593344] nix-daemon[840]: accepted connection from pid 1163, user nix-grpc-daemon (trusted) worker2 # [ 88.610091] nix-daemon[840]: reaped child process 1166, status = succeeded lb # [ 88.613746] niks3-server[962]: 2026/09/23 04:12:43 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 88.626339] nix-grpc-daemon[853]: time=2026-09-23T04:12:43.205Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 88.627880] nix-grpc-daemon[853]: time=2026-09-23T04:12:43.205Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)" lb # [ 88.643158] niks3-server[962]: 2026/09/23 04:12:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign worker2 # [ 88.654307] nix-grpc-daemon[853]: time=2026-09-23T04:12:43.233Z level=INFO msg="Uploading 1 narinfos" lb # [ 88.657963] niks3-server[962]: 2026/09/23 04:12:43 INFO Signed narinfos id=25 count=1 lb # [ 88.665899] niks3-server[962]: 2026/09/23 04:12:43 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls lb # [ 88.677397] niks3-server[962]: 2026/09/23 04:12:43 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst worker2 # [ 88.684606] nix-grpc-daemon[853]: time=2026-09-23T04:12:43.263Z level=INFO msg="Upload complete. (238ms)" lb # [ 88.685254] niks3-server[962]: 2026/09/23 04:12:43 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete lb # [ 88.689795] niks3-server[962]: 2026/09/23 04:12:43 INFO Completed upload id=25 worker2 # [ 88.692198] nix-grpc-daemon[771]: ts=2026-09-23T04:12:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 88.694796] niks3-server[962]: 2026/09/23 04:12:43 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo worker2 # [ 88.706099] nix-grpc-daemon[771]: ts=2026-09-23T04:12:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 88.714676] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 88.717820] nix-daemon[840]: reaped child process 1161, status = succeeded worker2 # [ 88.729299] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) lb # [ 89.890931] envoy[906]: conn peer=[2001:db8:1::1]:35780 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2990 tx=812 ms=4299 lb # [ 89.895849] envoy[906]: conn peer=[2001:db8:1::1]:35786 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1443 tx=1197 ms=4805 lb # [ 89.901036] envoy[906]: conn peer=[2001:db8:1::1]:44670 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=66 lb # [ 89.905728] envoy[906]: conn peer=[2001:db8:1::1]:44668 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3440 tx=1968 ms=2193 lb # [ 89.910467] envoy[906]: conn peer=[2001:db8:1::1]:44672 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3888 tx=2158 ms=2257 lb # [ 89.915989] envoy[906]: conn peer=[2001:db8:1::1]:44688 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=375 ms=70 worker2 # [ 90.396979] nix-daemon[840]: reaped child process 1170, status = succeeded worker2 # [ 90.511043] nix-daemon[840]: accepted connection from pid 1186, user nix-grpc-daemon (trusted) worker2 # [ 90.529902] nix-daemon[840]: reaped child process 1189, status = succeeded lb # [ 90.533473] niks3-server[962]: 2026/09/23 04:12:45 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 90.547841] nix-grpc-daemon[853]: time=2026-09-23T04:12:45.125Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 90.549722] nix-grpc-daemon[853]: time=2026-09-23T04:12:45.125Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)" lb # [ 90.575528] niks3-server[962]: 2026/09/23 04:12:45 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst worker2 # [ 90.587782] nix-grpc-daemon[853]: time=2026-09-23T04:12:45.166Z level=INFO msg="Uploading 1 narinfos" lb # [ 90.592200] niks3-server[962]: 2026/09/23 04:12:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign lb # [ 90.604123] niks3-server[962]: 2026/09/23 04:12:45 INFO Signed narinfos id=26 count=1 lb # [ 90.606609] niks3-server[962]: 2026/09/23 04:12:45 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv lb # [ 90.619464] niks3-server[962]: 2026/09/23 04:12:45 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls lb # [ 90.624938] niks3-server[962]: 2026/09/23 04:12:45 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete worker2 # [ 90.627294] nix-grpc-daemon[853]: time=2026-09-23T04:12:45.206Z level=INFO msg="Upload complete. (284ms)" lb # [ 90.628900] niks3-server[962]: 2026/09/23 04:12:45 INFO Completed upload id=26 lb # [ 90.632675] niks3-server[962]: 2026/09/23 04:12:45 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo worker2 # [ 90.641152] nix-grpc-daemon[771]: ts=2026-09-23T04:12:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=14 outputs=1 worker2 # [ 90.659353] nix-grpc-daemon[771]: ts=2026-09-23T04:12:45Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 90.665540] nix-daemon[840]: reaped child process 1168, status = succeeded worker2 # [ 90.689786] nix-grpc-daemon[771]: ts=2026-09-23T04:12:45Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 90.696345] nix-grpc-daemon[771]: ts=2026-09-23T04:12:45Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_out=232 client # [ 90.821824] nix-daemon[1150]: reaped child process 1152, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.80 seconds) client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.04 seconds) client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds) client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds) (finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.92 seconds) subtest: an input only one worker has still reaches the builder client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only client # this derivation will be built: client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.29 seconds) client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.12 seconds) worker1: must succeed: nix-store --import < /tmp/shared/inp.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.19 seconds) worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.03 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2 client # [ 91.779458] nix-daemon[1150]: accepted connection from pid 1232, user root (trusted) client # this derivation will be built: client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker2 # [ 91.926413] nix-grpc-daemon[771]: ts=2026-09-23T04:12:46Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:49370 worker1 # [ 91.926580] nix-grpc-daemon[766]: ts=2026-09-23T04:12:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 92.094874] nix-daemon[849]: accepted connection from pid 1130, user nix-grpc-daemon (trusted) worker1 # [ 92.114446] nix-daemon[849]: reaped child process 1133, status = succeeded lb # [ 92.115462] niks3-server[962]: 2026/09/23 04:12:46 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 92.125564] nix-grpc-daemon[902]: time=2026-09-23T04:12:46.727Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 92.127748] nix-grpc-daemon[902]: time=2026-09-23T04:12:46.727Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)" lb # [ 92.151669] niks3-server[962]: 2026/09/23 04:12:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign worker1 # [ 92.158277] nix-grpc-daemon[902]: time=2026-09-23T04:12:46.760Z level=INFO msg="Uploading 1 narinfos" lb # [ 92.165535] niks3-server[962]: 2026/09/23 04:12:46 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst lb # [ 92.175435] niks3-server[962]: 2026/09/23 04:12:46 INFO Signed narinfos id=27 count=1 lb # [ 92.182180] niks3-server[962]: 2026/09/23 04:12:46 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls worker1 # [ 92.186360] nix-grpc-daemon[902]: time=2026-09-23T04:12:46.788Z level=INFO msg="Upload complete. (254ms)" lb # [ 92.190666] niks3-server[962]: 2026/09/23 04:12:46 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete lb # [ 92.195877] niks3-server[962]: 2026/09/23 04:12:46 INFO Completed upload id=27 lb # [ 92.199923] niks3-server[962]: 2026/09/23 04:12:46 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo worker1 # [ 92.210131] nix-grpc-daemon[766]: ts=2026-09-23T04:12:46Z level=debug event=want client=14 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=0 worker1 # [ 92.217508] nix-grpc-daemon[766]: ts=2026-09-23T04:12:46Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 92.255255] nix-grpc-daemon[766]: ts=2026-09-23T04:12:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 92.270640] nix-grpc-daemon[766]: ts=2026-09-23T04:12:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 client # copying 2 paths to 'lb:50051'... client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'... worker1 # [ 92.290253] nix-grpc-daemon[766]: ts=2026-09-23T04:12:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'... worker1 # [ 92.304129] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 92.460206] nix-daemon[849]: accepted connection from pid 1139, user nix-grpc-daemon (trusted) worker1 # [ 92.479062] nix-daemon[849]: reaped child process 1142, status = succeeded lb # [ 92.478141] niks3-server[962]: 2026/09/23 04:12:47 INFO Received uploads request method=POST path=/api/pending_closures lb # [ 92.488267] niks3-server[962]: 2026/09/23 04:12:47 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 92.500076] nix-grpc-daemon[902]: time=2026-09-23T04:12:47.100Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker1 # [ 92.501813] nix-grpc-daemon[902]: time=2026-09-23T04:12:47.100Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)" worker1 # [ 92.503899] nix-grpc-daemon[902]: time=2026-09-23T04:12:47.100Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)" lb # [ 92.528632] niks3-server[962]: 2026/09/23 04:12:47 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst lb # [ 92.539230] niks3-server[962]: 2026/09/23 04:12:47 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst lb # [ 92.551233] niks3-server[962]: 2026/09/23 04:12:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign lb # [ 92.557695] niks3-server[962]: 2026/09/23 04:12:47 INFO Signed narinfos id=29 count=2 worker1 # [ 92.555885] nix-grpc-daemon[902]: time=2026-09-23T04:12:47.157Z level=INFO msg="Uploading 3 narinfos" lb # [ 92.562918] niks3-server[962]: 2026/09/23 04:12:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign lb # [ 92.569341] niks3-server[962]: 2026/09/23 04:12:47 INFO Signed narinfos id=28 count=1 lb # [ 92.573240] niks3-server[962]: 2026/09/23 04:12:47 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls lb # [ 92.575792] niks3-server[962]: 2026/09/23 04:12:47 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls lb # [ 92.593010] niks3-server[962]: 2026/09/23 04:12:47 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete lb # [ 92.600749] niks3-server[962]: 2026/09/23 04:12:47 INFO Completed upload id=28 lb # [ 92.604469] niks3-server[962]: 2026/09/23 04:12:47 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete lb # [ 92.608863] niks3-server[962]: 2026/09/23 04:12:47 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo worker1 # [ 92.606372] nix-grpc-daemon[902]: time=2026-09-23T04:12:47.208Z level=INFO msg="Upload complete. (284ms)" lb # [ 92.611903] niks3-server[962]: 2026/09/23 04:12:47 INFO Completed upload id=29 lb # [ 92.614665] niks3-server[962]: 2026/09/23 04:12:47 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo lb # [ 92.618184] niks3-server[962]: 2026/09/23 04:12:47 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo worker1 # [ 92.617562] nix-grpc-daemon[766]: ts=2026-09-23T04:12:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=0 paths=2 nar_bytes_in=984 worker1 # [ 92.635517] nix-grpc-daemon[766]: ts=2026-09-23T04:12:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 92.646592] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 92.654572] nix-daemon[849]: reaped child process 1137, status = succeeded client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051... worker1 # [ 92.666858] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) client # uses-input-w1onlya> node-a: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 94.170445] nix-daemon[849]: reaped child process 1146, status = succeeded worker1 # [ 94.254371] nix-daemon[849]: accepted connection from pid 1162, user nix-grpc-daemon (trusted) worker1 # [ 94.272102] nix-daemon[849]: reaped child process 1165, status = succeeded lb # [ 94.279341] niks3-server[962]: 2026/09/23 04:12:48 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 94.288172] nix-grpc-daemon[902]: time=2026-09-23T04:12:48.889Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 94.309665] niks3-server[962]: 2026/09/23 04:12:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign worker1 # [ 94.316463] nix-grpc-daemon[902]: time=2026-09-23T04:12:48.918Z level=INFO msg="Uploading 1 narinfos" lb # [ 94.324220] niks3-server[962]: 2026/09/23 04:12:48 INFO Signed narinfos id=30 count=1 lb # [ 94.331339] niks3-server[962]: 2026/09/23 04:12:48 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv lb # [ 94.345649] niks3-server[962]: 2026/09/23 04:12:48 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls worker1 # [ 94.349571] nix-grpc-daemon[902]: time=2026-09-23T04:12:48.952Z level=INFO msg="Upload complete. (266ms)" lb # [ 94.351270] niks3-server[962]: 2026/09/23 04:12:48 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete lb # [ 94.355583] niks3-server[962]: 2026/09/23 04:12:48 INFO Completed upload id=30 lb # [ 94.362362] niks3-server[962]: 2026/09/23 04:12:48 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo worker1 # [ 94.360916] nix-grpc-daemon[766]: ts=2026-09-23T04:12:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=15 outputs=1 worker1 # [ 94.376771] nix-daemon[849]: reaped child process 1144, status = succeeded client # copying 1 paths... worker1 # [ 94.406980] nix-grpc-daemon[766]: ts=2026-09-23T04:12:49Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:55052 client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'... worker1 # [ 94.440209] nix-grpc-daemon[766]: ts=2026-09-23T04:12:49Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 94.447035] nix-grpc-daemon[766]: ts=2026-09-23T04:12:49Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=0 paths=1 nar_bytes_out=120 client # [ 94.578707] nix-daemon[1150]: reaped child process 1235, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 3.00 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2 client # [ 94.739731] nix-daemon[1150]: accepted connection from pid 1253, user root (trusted) client # this derivation will be built: client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker2 # [ 94.887836] nix-grpc-daemon[771]: ts=2026-09-23T04:12:49Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:49370 worker1 # [ 94.886842] nix-grpc-daemon[766]: ts=2026-09-23T04:12:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 94.896667] nix-grpc-daemon[902]: time=2026-09-23T04:12:49.499Z level=INFO msg="All 1 paths already cached" worker1 # [ 94.913462] nix-grpc-daemon[766]: ts=2026-09-23T04:12:49Z level=debug event=want client=15 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=0 worker1 # [ 94.920450] nix-grpc-daemon[766]: ts=2026-09-23T04:12:49Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 94.927118] nix-grpc-daemon[771]: ts=2026-09-23T04:12:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 94.942682] nix-grpc-daemon[771]: ts=2026-09-23T04:12:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49370 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'... worker2 # [ 94.962366] nix-grpc-daemon[771]: ts=2026-09-23T04:12:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 94.977998] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 95.000817] nix-grpc-daemon[771]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 52 ms (attempt 1/5) worker2 # [ 95.060882] nix-grpc-daemon[771]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 163 ms (attempt 2/5) worker2 # [ 95.225976] nix-grpc-daemon[771]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 343 ms (attempt 3/5) worker2 # [ 95.570844] nix-grpc-daemon[771]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 332 ms (attempt 4/5) worker2 # [ 95.914511] nix-grpc-daemon[771]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 95.921507] nix-grpc-daemon[771]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 95.928899] nix-grpc-daemon[771]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'... worker2 # [ 96.107111] nix-daemon[840]: accepted connection from pid 1201, user nix-grpc-daemon (trusted) worker2 # [ 96.126204] nix-daemon[840]: reaped child process 1204, status = succeeded lb # [ 96.131983] niks3-server[962]: 2026/09/23 04:12:50 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 96.146615] nix-grpc-daemon[853]: time=2026-09-23T04:12:50.725Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 96.148491] nix-grpc-daemon[853]: time=2026-09-23T04:12:50.725Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)" lb # [ 96.166788] niks3-server[962]: 2026/09/23 04:12:50 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls worker2 # [ 96.178581] nix-grpc-daemon[853]: time=2026-09-23T04:12:50.757Z level=INFO msg="Uploading 1 narinfos" lb # [ 96.178059] niks3-server[962]: 2026/09/23 04:12:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign lb # [ 96.189925] niks3-server[962]: 2026/09/23 04:12:50 INFO Signed narinfos id=31 count=1 lb # [ 96.195961] niks3-server[962]: 2026/09/23 04:12:50 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst worker2 # [ 96.208679] nix-grpc-daemon[853]: time=2026-09-23T04:12:50.787Z level=INFO msg="Upload complete. (260ms)" lb # [ 96.206481] niks3-server[962]: 2026/09/23 04:12:50 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete lb # [ 96.211528] niks3-server[962]: 2026/09/23 04:12:50 INFO Completed upload id=31 lb # [ 96.214330] niks3-server[962]: 2026/09/23 04:12:50 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo worker2 # [ 96.220326] nix-grpc-daemon[771]: ts=2026-09-23T04:12:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=1 paths=1 nar_bytes_in=568 worker2 # [ 96.232747] nix-grpc-daemon[771]: ts=2026-09-23T04:12:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 96.245491] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) worker2 # [ 96.251483] nix-daemon[840]: reaped child process 1194, status = succeeded worker2 # [ 96.266376] nix-grpc-daemon[771]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'... client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051... worker2 # [ 96.282938] nix-daemon[840]: accepted connection from pid 771, user nix-grpc-daemon (trusted) client # uses-input-w1onlyb> worker2: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 97.533767] nix-daemon[840]: reaped child process 1209, status = succeeded worker2 # [ 97.650392] nix-daemon[840]: accepted connection from pid 1225, user nix-grpc-daemon (trusted) worker2 # [ 97.667206] nix-daemon[840]: reaped child process 1228, status = succeeded lb # [ 97.674367] niks3-server[962]: 2026/09/23 04:12:52 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 97.688385] nix-grpc-daemon[853]: time=2026-09-23T04:12:52.266Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 97.704556] niks3-server[962]: 2026/09/23 04:12:52 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls lb # [ 97.716384] niks3-server[962]: 2026/09/23 04:12:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign worker2 # [ 97.724723] nix-grpc-daemon[853]: time=2026-09-23T04:12:52.303Z level=INFO msg="Uploading 1 narinfos" lb # [ 97.728265] niks3-server[962]: 2026/09/23 04:12:52 INFO Signed narinfos id=32 count=1 lb # [ 97.734192] niks3-server[962]: 2026/09/23 04:12:52 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker2 # [ 97.749607] nix-grpc-daemon[853]: time=2026-09-23T04:12:52.328Z level=INFO msg="Upload complete. (251ms)" lb # [ 97.747012] niks3-server[962]: 2026/09/23 04:12:52 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete lb # [ 97.751024] niks3-server[962]: 2026/09/23 04:12:52 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo lb # [ 97.756643] niks3-server[962]: 2026/09/23 04:12:52 INFO Completed upload id=32 client # copying 1 paths... worker1 # [ 97.766172] nix-grpc-daemon[766]: ts=2026-09-23T04:12:52Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:55052 worker2 # [ 97.761028] nix-grpc-daemon[771]: ts=2026-09-23T04:12:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=16 outputs=1 worker2 # [ 97.777206] nix-daemon[840]: reaped child process 1206, status = succeeded worker1 # [ 97.778446] nix-grpc-daemon[766]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'... client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'... worker2 # [ 97.827332] nix-grpc-daemon[771]: ts=2026-09-23T04:12:52Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 97.832921] nix-grpc-daemon[771]: ts=2026-09-23T04:12:52Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:49370 duration_s=0 paths=1 nar_bytes_out=120 client # [ 97.908918] nix-daemon[1150]: reaped child process 1256, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.33 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2 client # [ 98.080176] nix-daemon[1150]: accepted connection from pid 1275, user root (trusted) client # this derivation will be built: client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker1 # [ 98.218764] nix-grpc-daemon[766]: ts=2026-09-23T04:12:52Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:55052 client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 98.226497] nix-grpc-daemon[766]: ts=2026-09-23T04:12:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 98.234472] nix-grpc-daemon[902]: time=2026-09-23T04:12:52.834Z level=INFO msg="All 1 paths already cached" worker1 # [ 98.245942] nix-grpc-daemon[766]: ts=2026-09-23T04:12:52Z level=debug event=want client=16 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=0 worker1 # [ 98.255289] nix-grpc-daemon[766]: ts=2026-09-23T04:12:52Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 98.265181] nix-grpc-daemon[766]: ts=2026-09-23T04:12:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 98.272680] nix-grpc-daemon[766]: ts=2026-09-23T04:12:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 98.282526] nix-grpc-daemon[902]: time=2026-09-23T04:12:52.885Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'... worker1 # [ 98.293336] nix-grpc-daemon[766]: ts=2026-09-23T04:12:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 98.308120] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 98.455543] nix-daemon[849]: accepted connection from pid 1174, user nix-grpc-daemon (trusted) worker1 # [ 98.472994] nix-daemon[849]: reaped child process 1177, status = succeeded lb # [ 98.478923] niks3-server[962]: 2026/09/23 04:12:53 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 98.487031] nix-grpc-daemon[902]: time=2026-09-23T04:12:53.089Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 98.488562] nix-grpc-daemon[902]: time=2026-09-23T04:12:53.089Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)" lb # [ 98.508849] niks3-server[962]: 2026/09/23 04:12:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign worker1 # [ 98.513635] nix-grpc-daemon[902]: time=2026-09-23T04:12:53.116Z level=INFO msg="Uploading 1 narinfos" lb # [ 98.521571] niks3-server[962]: 2026/09/23 04:12:53 INFO Signed narinfos id=33 count=1 lb # [ 98.530094] niks3-server[962]: 2026/09/23 04:12:53 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst lb # [ 98.538117] niks3-server[962]: 2026/09/23 04:12:53 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls worker1 # [ 98.546366] nix-grpc-daemon[902]: time=2026-09-23T04:12:53.148Z level=INFO msg="Upload complete. (224ms)" lb # [ 98.548180] niks3-server[962]: 2026/09/23 04:12:53 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete lb # [ 98.553270] niks3-server[962]: 2026/09/23 04:12:53 INFO Completed upload id=33 worker1 # [ 98.556270] nix-grpc-daemon[766]: ts=2026-09-23T04:12:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=0 paths=1 nar_bytes_in=568 lb # [ 98.560426] niks3-server[962]: 2026/09/23 04:12:53 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo worker1 # [ 98.567940] nix-grpc-daemon[766]: ts=2026-09-23T04:12:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 98.577444] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) worker1 # [ 98.580355] nix-daemon[849]: reaped child process 1172, status = succeeded client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051... worker1 # [ 98.592949] nix-daemon[849]: accepted connection from pid 766, user nix-grpc-daemon (trusted) client # uses-input-w1onlyc> node-a: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds lb # [ 99.897365] envoy[906]: conn peer=[2001:db8:1::1]:44710 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=406 ms=71 lb # [ 99.902226] envoy[906]: conn peer=[2001:db8:1::1]:44704 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4352 tx=2416 ms=2432 lb # [ 99.907292] envoy[906]: conn peer=[2001:db8:1::1]:58078 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3918 tx=2084 ms=2581 lb # [ 99.912278] envoy[906]: conn peer=[2001:db8:1::1]:58080 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=67 lb # [ 99.916980] envoy[906]: conn peer=[2001:db8:1::1]:58096 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3740 tx=2124 ms=2957 lb # [ 99.922582] envoy[906]: conn peer=[2001:db8:1::1]:58102 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=422 tx=328 ms=23 worker1 # [ 100.029201] nix-daemon[849]: reaped child process 1181, status = succeeded worker1 # [ 100.149482] nix-daemon[849]: accepted connection from pid 1197, user nix-grpc-daemon (trusted) worker1 # [ 100.169502] nix-daemon[849]: reaped child process 1200, status = succeeded lb # [ 100.175933] niks3-server[962]: 2026/09/23 04:12:54 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 100.184410] nix-grpc-daemon[902]: time=2026-09-23T04:12:54.785Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 100.207734] niks3-server[962]: 2026/09/23 04:12:54 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls worker1 # [ 100.219028] nix-grpc-daemon[902]: time=2026-09-23T04:12:54.821Z level=INFO msg="Uploading 1 narinfos" lb # [ 100.222089] niks3-server[962]: 2026/09/23 04:12:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign lb # [ 100.231943] niks3-server[962]: 2026/09/23 04:12:54 INFO Signed narinfos id=34 count=1 worker1 # [ 100.245283] nix-grpc-daemon[902]: time=2026-09-23T04:12:54.847Z level=INFO msg="Upload complete. (243ms)" lb # [ 100.241105] niks3-server[962]: 2026/09/23 04:12:54 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv lb # [ 100.252939] niks3-server[962]: 2026/09/23 04:12:54 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete lb # [ 100.257918] niks3-server[962]: 2026/09/23 04:12:54 INFO Completed upload id=34 worker1 # [ 100.256939] nix-grpc-daemon[766]: ts=2026-09-23T04:12:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=17 outputs=1 lb # [ 100.260974] niks3-server[962]: 2026/09/23 04:12:54 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo client # copying 1 paths... worker1 # [ 100.269771] nix-daemon[849]: reaped child process 1179, status = succeeded worker2 # [ 100.275479] nix-grpc-daemon[771]: ts=2026-09-23T04:12:54Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:49370 worker2 # [ 100.288827] nix-grpc-daemon[771]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'... client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'... worker1 # [ 100.326856] nix-grpc-daemon[766]: ts=2026-09-23T04:12:54Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 100.334223] nix-grpc-daemon[766]: ts=2026-09-23T04:12:54Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55052 duration_s=0 paths=1 nar_bytes_out=120 client # [ 100.421642] nix-daemon[1150]: reaped child process 1278, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.51 seconds) (finished: subtest: an input only one worker has still reaches the builder, in 9.47 seconds) subtest: upload whose reference the worker lacks is completed from the cache client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache client # this derivation will be built: client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.33 seconds) client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache client # this derivation will be built: client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.27 seconds) client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.12 seconds) worker1: must succeed: nix-store --import < /tmp/shared/ref.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.19 seconds) worker2 # [ 101.407128] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 102.267054] nix-grpc-daemon[771]: ts=2026-09-23T04:12:56Z level=info event=signal_exit worker2 # [ 103.441240] nix-grpc-daemon[771]: ts=2026-09-23T04:12:58Z level=info event=scheduler_restarting worker1 # [ 103.448468] nix-grpc-daemon[766]: ts=2026-09-23T04:12:58Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:55038 worker1 # [ 103.448758] nix-grpc-daemon[766]: ts=2026-09-23T04:12:58Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 103.478681] nix-grpc-daemon[771]: ts=2026-09-23T04:12:58Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 103.508524] nix-daemon[840]: reaped child process 842, status = succeeded worker2 # [ 104.234945] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 104.239861] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 104.247872] systemd[1]: nix-grpc-daemon.service: Consumed 3.602s CPU time over 1min 24.207s wall clock time, 65.4M memory peak, 99K incoming IP traffic, 219.3K outgoing IP traffic. worker2 # [ 104.261803] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker2 # [ 104.272173] systemd[1]: Closed Nix worker-protocol over gRPC. worker2 # [ 104.273974] systemd[1]: nix-grpc-daemon.socket: Consumed 557us CPU time over 1min 34.158s wall clock time, 256K memory peak, 98.7K incoming IP traffic, 87.8K outgoing IP traffic. lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds) client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2 worker1 # [ 104.555457] nix-grpc-daemon[766]: ts=2026-09-23T04:12:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55052 worker1 # [ 104.564468] nix-grpc-daemon[766]: ts=2026-09-23T04:12:59Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:55052 client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.25 seconds) worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds) worker2 # [ 104.695026] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 104.762005] systemd[1]: Stopping Nix worker-protocol over gRPC... worker1 # [ 105.143210] nix-grpc-daemon[766]: ts=2026-09-23T04:12:59Z level=info event=signal_exit worker2 # [ 105.433111] systemd[1]: Starting scheduler-token.service... lb # [ 105.443224] niks3-server[962]: 2026/09/23 04:13:00 INFO lead: released remote=192.168.1.3:57068 worker2 # [ 105.522774] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 105.529459] systemd[1]: Finished scheduler-token.service. worker2 # [ 105.531623] systemd[1]: scheduler-token.service: Consumed 61ms CPU time over 94ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 105.556237] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 105.670721] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:00Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 105.673108] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:00Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 105.677082] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:00Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 105.684048] nix-grpc-daemon[1253]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 105.685535] nix-grpc-daemon[1253]: E0000 00:00:1790136780.260981 1253 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 105.687858] nix-grpc-daemon[1253]: E0000 00:00:1790136780.263127 1253 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb # [ 105.700191] niks3-server[962]: 2026/09/23 04:13:00 INFO lead: acquired remote=192.168.1.4:53894 worker2 # [ 105.735130] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:00Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 105.737434] nix-grpc-daemon[1253]: E0000 00:00:1790136780.280850 1296 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 105.739989] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:00Z level=info event=scheduler_take_over worker2 # [ 105.741159] nix-grpc-daemon[1253]: E0000 00:00:1790136780.298995 1296 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 105.746182] nix-grpc-daemon[1253]: E0000 00:00:1790136780.299154 1296 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 105.751007] nix-grpc-daemon[1253]: E0000 00:00:1790136780.313499 1296 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 105.753551] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 105.754369] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 105.755789] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 105.952576] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 105.957591] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 106.158631] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 106.164107] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 106.364701] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 106.369765] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 106.445409] nix-grpc-daemon[766]: ts=2026-09-23T04:13:01Z level=info event=scheduler_restarting worker1 # [ 106.452377] nix-grpc-daemon[766]: ts=2026-09-23T04:13:01Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 106.456978] nix-grpc-daemon[766]: ts=2026-09-23T04:13:01Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:55038 worker1 # [ 106.462198] nix-grpc-daemon[766]: ts=2026-09-23T04:13:01Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 106.490444] nix-daemon[849]: reaped child process 851, status = succeeded worker2 # [ 106.570753] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 106.576277] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 106.668608] nix-grpc-daemon[766]: ts=2026-09-23T04:13:01Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 106.778253] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 106.795180] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:01Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 106.799829] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:01Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:49954 worker2 # [ 106.805351] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:01Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 107.257678] nix-grpc-daemon[766]: E0000 00:00:1790136781.859908 812 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true worker1 # [ 107.286563] nix-grpc-daemon[766]: E0000 00:00:1790136781.888994 812 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker1 # [ 108.178553] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 108.184960] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 108.187611] systemd[1]: nix-grpc-daemon.service: Consumed 2.897s CPU time over 1min 28.292s wall clock time, 65M memory peak, 95.2K incoming IP traffic, 184.3K outgoing IP traffic. lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 108.210804] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker1 # [ 108.212974] systemd[1]: Closed Nix worker-protocol over gRPC. worker1 # [ 108.214466] systemd[1]: nix-grpc-daemon.socket: Consumed 544us CPU time over 1min 38.097s wall clock time, 256K memory peak, 120.1K incoming IP traffic, 104.3K outgoing IP traffic. lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds) client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2 worker2 # [ 108.506843] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49964 worker2 # [ 108.522842] nix-daemon[840]: accepted connection from pid 1253, user nix-grpc-daemon (trusted) worker2 # [ 108.549057] nix-grpc-daemon[1253]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 72 ms (attempt 1/5) worker2 # [ 108.629350] nix-grpc-daemon[1253]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 173 ms (attempt 2/5) worker2 # [ 108.804215] nix-grpc-daemon[1253]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 331 ms (attempt 3/5) worker2 # [ 109.137350] nix-grpc-daemon[1253]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 379 ms (attempt 4/5) worker2 # [ 109.522928] nix-grpc-daemon[1253]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 109.529945] nix-grpc-daemon[1253]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org client # copying 2 paths... client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'... worker2 # [ 109.555415] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49964 client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'... worker2 # [ 109.570552] nix-daemon[840]: accepted connection from pid 1253, user nix-grpc-daemon (trusted) worker2 # [ 109.610358] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:04Z level=info event=niks3_push_started pid=1314 worker2 # [ 109.763228] nix-grpc-daemon[1314]: time=2026-09-23T04:13:04.342Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 109.874499] nix-daemon[840]: accepted connection from pid 1326, user nix-grpc-daemon (trusted) worker2 # [ 109.893750] nix-daemon[840]: reaped child process 1329, status = succeeded lb # [ 109.897188] envoy[906]: conn peer=[2001:db8:1::1]:58114 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3724 tx=2107 ms=2131 lb # [ 109.902798] envoy[906]: conn peer=[2001:db8:1::1]:58118 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=26 lb # [ 109.910308] envoy[906]: conn peer=[2001:db8:1::4]:52356 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=51034 tx=3229 ms=83171 lb # [ 109.919583] envoy[906]: conn peer=[2001:db8:1::1]:60398 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=30 lb # [ 109.925004] envoy[906]: conn peer=[2001:db8:1::3]:43230 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=10634 tx=3071 ms=86476 worker2 # [ 109.936068] nix-grpc-daemon[1314]: time=2026-09-23T04:13:04.514Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker2 # [ 109.937832] nix-grpc-daemon[1314]: time=2026-09-23T04:13:04.514Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)" lb # [ 109.934266] niks3-server[962]: 2026/09/23 04:13:04 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 109.939820] nix-grpc-daemon[1314]: time=2026-09-23T04:13:04.514Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)" lb # [ 109.938745] niks3-server[962]: 2026/09/23 04:13:04 INFO Received uploads request method=POST path=/api/pending_closures lb # [ 109.976939] niks3-server[962]: 2026/09/23 04:13:04 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst lb # [ 109.988394] niks3-server[962]: 2026/09/23 04:13:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign lb # [ 110.002069] niks3-server[962]: 2026/09/23 04:13:04 INFO Signed narinfos id=35 count=1 worker2 # [ 110.004109] nix-grpc-daemon[1314]: time=2026-09-23T04:13:04.580Z level=INFO msg="Uploading 3 narinfos" lb # [ 110.003716] niks3-server[962]: 2026/09/23 04:13:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign lb # [ 110.013475] niks3-server[962]: 2026/09/23 04:13:04 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst lb # [ 110.022756] niks3-server[962]: 2026/09/23 04:13:04 INFO Signed narinfos id=36 count=2 lb # [ 110.028757] niks3-server[962]: 2026/09/23 04:13:04 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls lb # [ 110.035821] niks3-server[962]: 2026/09/23 04:13:04 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls lb # [ 110.043191] niks3-server[962]: 2026/09/23 04:13:04 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo lb # [ 110.054897] niks3-server[962]: 2026/09/23 04:13:04 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete lb # [ 110.061781] niks3-server[962]: 2026/09/23 04:13:04 INFO Completed upload id=35 lb # [ 110.065948] niks3-server[962]: 2026/09/23 04:13:04 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete lb # [ 110.070505] niks3-server[962]: 2026/09/23 04:13:04 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo lb # [ 110.072981] niks3-server[962]: 2026/09/23 04:13:04 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo lb # [ 110.075987] niks3-server[962]: 2026/09/23 04:13:04 INFO Completed upload id=36 worker2 # [ 110.078541] nix-grpc-daemon[1314]: time=2026-09-23T04:13:04.657Z level=INFO msg="Upload complete. (311ms)" worker2 # [ 110.089153] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49964 duration_s=0 paths=2 nar_bytes_in=312 worker2 # [ 110.108421] nix-daemon[840]: reaped child process 1312, status = succeeded client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.84 seconds) worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds) worker1 # [ 110.235255] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 110.245925] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 110.399053] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:05Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 110.404611] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:05Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 110.406196] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:05Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 110.412053] nix-grpc-daemon[1223]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 110.413575] nix-grpc-daemon[1223]: E0000 00:00:1790136785.013074 1223 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 110.456128] nix-grpc-daemon[1223]: E0000 00:00:1790136785.019850 1223 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 110.458546] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:05Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 110.465335] nix-grpc-daemon[1223]: E0000 00:00:1790136785.029913 1263 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 110.468265] nix-grpc-daemon[1223]: E0000 00:00:1790136785.046621 1263 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 110.470693] nix-grpc-daemon[1223]: E0000 00:00:1790136785.047814 1263 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 110.473137] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 110.473933] nix-grpc-daemon[1223]: E0000 00:00:1790136785.066901 1263 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 110.482190] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:05Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 110.487086] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:05Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:49954 worker2 # [ 110.494076] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:05Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.14 seconds) (finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.15 seconds) subtest: a niks3 restart does not cost the leader its role lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb # [ 111.700399] systemd[1]: Stopping niks3 server... worker1 # [ 111.700262] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:06Z level=info event=scheduler_lock_lost status=200 lb # [ 111.706504] niks3-server[962]: 2026/09/23 04:13:06 INFO Shutdown signal received, draining in-flight requests timeout=10s worker2 # [ 111.713871] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:06Z level=info event=scheduler_lock_lost status=200 lb # [ 111.724132] niks3-server[962]: 2026/09/23 04:13:06 INFO lead: released remote=192.168.1.4:53894 lb # [ 111.730519] systemd[1]: niks3.service: Deactivated successfully. lb # [ 111.733066] systemd[1]: Stopped niks3 server. lb # [ 111.733785] systemd[1]: niks3.service: Consumed 639ms CPU time over 1min 29.521s wall clock time, 27.4M memory peak, 93.6K incoming IP traffic, 136.3K outgoing IP traffic. lb # [ 111.743296] systemd[1]: Starting niks3 server... lb # [ 111.802653] niks3-server[1102]: 2026/09/23 04:13:06 goose: no migrations to run. current version: 20260920000000 lb # [ 111.807465] niks3-server[1102]: 2026/09/23 04:13:06 OK 1_commit_pending_closure.sql (6.22ms) lb # [ 111.811145] niks3-server[1102]: 2026/09/23 04:13:06 OK 2_object_stats_trigger.sql (3.32ms) lb # [ 111.812915] niks3-server[1102]: 2026/09/23 04:13:06 goose: up to current file version: 2 lb # [ 111.814583] niks3-server[1102]: 2026/09/23 04:13:06 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key lb # [ 111.824580] niks3-server[1102]: 2026/09/23 04:13:06 INFO Read proxy enabled — serving cache objects from S3 lb # [ 111.825910] niks3-server[1102]: 2026/09/23 04:13:06 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 111.830600] niks3-server[1102]: 2026/09/23 04:13:06 INFO systemd watchdog enabled interval=15s lb # [ 111.831993] niks3-server[1102]: 2026/09/23 04:13:06 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 111.834317] systemd[1]: Started niks3 server. lb: waiting for unit niks3.service lb: (finished: waiting for unit niks3.service, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.06 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2 worker2 # [ 112.309230] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:49964 client # don't know how to build these paths: client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker2 # [ 112.319319] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49964 worker2 # [ 112.350212] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:06Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=0 worker2 # [ 112.357661] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:06Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 112.395243] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49964 worker2 # [ 112.410799] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49964 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'... worker2 # [ 112.427047] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49964 worker2 # [ 112.442168] nix-daemon[840]: accepted connection from pid 1253, user nix-grpc-daemon (trusted) worker2 # [ 112.601275] nix-daemon[840]: accepted connection from pid 1346, user nix-grpc-daemon (trusted) worker2 # [ 112.619946] nix-daemon[840]: reaped child process 1349, status = succeeded lb # [ 112.620612] niks3-server[1102]: 2026/09/23 04:13:07 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 112.637089] nix-grpc-daemon[1314]: time=2026-09-23T04:13:07.215Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 112.638847] nix-grpc-daemon[1314]: time=2026-09-23T04:13:07.215Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)" lb # [ 112.663082] niks3-server[1102]: 2026/09/23 04:13:07 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls lb # [ 112.675123] niks3-server[1102]: 2026/09/23 04:13:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign lb # [ 112.690255] niks3-server[1102]: 2026/09/23 04:13:07 INFO Signed narinfos id=37 count=1 worker2 # [ 112.693619] nix-grpc-daemon[1314]: time=2026-09-23T04:13:07.271Z level=INFO msg="Uploading 1 narinfos" lb # [ 112.705005] niks3-server[1102]: 2026/09/23 04:13:07 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst lb # [ 112.719566] niks3-server[1102]: 2026/09/23 04:13:07 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete lb # [ 112.733518] niks3-server[1102]: 2026/09/23 04:13:07 INFO lead: acquired remote=192.168.1.4:56566 lb # [ 112.763873] niks3-server[1102]: 2026/09/23 04:13:07 INFO Completed upload id=37 worker2 # [ 112.766930] nix-grpc-daemon[1314]: time=2026-09-23T04:13:07.346Z level=INFO msg="Upload complete. (307ms)" lb # [ 112.769107] niks3-server[1102]: 2026/09/23 04:13:07 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo worker2 # [ 112.778464] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49964 duration_s=0 paths=1 nar_bytes_in=544 worker2 # [ 112.796435] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49964 worker2 # [ 112.809147] nix-daemon[840]: accepted connection from pid 1253, user nix-grpc-daemon (trusted) worker2 # [ 112.812543] nix-daemon[840]: reaped child process 1344, status = succeeded client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051... worker2 # [ 112.826173] nix-daemon[840]: accepted connection from pid 1253, user nix-grpc-daemon (trusted) client # farm-a> worker2: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-after-niks3-restart worker2 # [ 114.499538] nix-daemon[840]: reaped child process 1355, status = succeeded worker2 # [ 114.616990] nix-daemon[840]: accepted connection from pid 1371, user nix-grpc-daemon (trusted) worker2 # [ 114.635390] nix-daemon[840]: reaped child process 1374, status = succeeded lb # [ 114.637196] niks3-server[1102]: 2026/09/23 04:13:09 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 114.654859] nix-grpc-daemon[1314]: time=2026-09-23T04:13:09.231Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 114.656833] nix-grpc-daemon[1314]: time=2026-09-23T04:13:09.231Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)" lb # [ 114.684762] niks3-server[1102]: 2026/09/23 04:13:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign worker2 # [ 114.698068] nix-grpc-daemon[1314]: time=2026-09-23T04:13:09.276Z level=INFO msg="Uploading 1 narinfos" lb # [ 114.699315] niks3-server[1102]: 2026/09/23 04:13:09 INFO Signed narinfos id=38 count=1 lb # [ 114.710961] niks3-server[1102]: 2026/09/23 04:13:09 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst lb # [ 114.724969] niks3-server[1102]: 2026/09/23 04:13:09 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls lb # [ 114.732156] niks3-server[1102]: 2026/09/23 04:13:09 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv lb # [ 114.745971] niks3-server[1102]: 2026/09/23 04:13:09 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete lb # [ 114.753391] niks3-server[1102]: 2026/09/23 04:13:09 INFO Completed upload id=38 worker2 # [ 114.755653] nix-grpc-daemon[1314]: time=2026-09-23T04:13:09.334Z level=INFO msg="Upload complete. (313ms)" lb # [ 114.758323] niks3-server[1102]: 2026/09/23 04:13:09 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo worker2 # [ 114.778412] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49964 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=1 worker2 # [ 114.798026] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:09Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=0 worker2 # [ 114.802802] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:09Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 114.807792] nix-daemon[840]: reaped child process 1353, status = succeeded worker2 # [ 114.822990] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49964 worker2 # [ 114.838300] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49964 worker2 # [ 114.853676] nix-grpc-daemon[1314]: time=2026-09-23T04:13:09.432Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'... worker2 # [ 114.864321] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49964 worker2 # [ 114.877593] nix-daemon[840]: accepted connection from pid 1253, user nix-grpc-daemon (trusted) worker2 # [ 115.030370] nix-daemon[840]: accepted connection from pid 1380, user nix-grpc-daemon (trusted) worker2 # [ 115.049209] nix-daemon[840]: reaped child process 1383, status = succeeded lb # [ 115.052781] niks3-server[1102]: 2026/09/23 04:13:09 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 115.074418] nix-grpc-daemon[1314]: time=2026-09-23T04:13:09.653Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 115.076466] nix-grpc-daemon[1314]: time=2026-09-23T04:13:09.653Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)" lb # [ 115.109144] niks3-server[1102]: 2026/09/23 04:13:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign lb # [ 115.120775] niks3-server[1102]: 2026/09/23 04:13:09 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst lb # [ 115.132428] niks3-server[1102]: 2026/09/23 04:13:09 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls lb # [ 115.145669] niks3-server[1102]: 2026/09/23 04:13:09 INFO Signed narinfos id=39 count=1 worker2 # [ 115.149286] nix-grpc-daemon[1314]: time=2026-09-23T04:13:09.728Z level=INFO msg="Uploading 1 narinfos" lb # [ 115.157914] niks3-server[1102]: 2026/09/23 04:13:09 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete lb # [ 115.168411] niks3-server[1102]: 2026/09/23 04:13:09 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo lb # [ 115.181378] niks3-server[1102]: 2026/09/23 04:13:09 INFO Completed upload id=39 worker2 # [ 115.184393] nix-grpc-daemon[1314]: time=2026-09-23T04:13:09.763Z level=INFO msg="Upload complete. (290ms)" worker2 # [ 115.196309] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49964 duration_s=0 paths=1 nar_bytes_in=656 worker2 # [ 115.209058] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49964 worker2 # [ 115.223592] nix-daemon[840]: accepted connection from pid 1253, user nix-grpc-daemon (trusted) worker2 # [ 115.233404] nix-daemon[840]: reaped child process 1378, status = succeeded client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051... client # farm-b> worker2: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker2 # [ 115.249940] nix-daemon[840]: accepted connection from pid 1253, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-after-niks3-restart worker2 # [ 116.282078] nix-daemon[840]: reaped child process 1387, status = succeeded worker2 # [ 116.380239] nix-daemon[840]: accepted connection from pid 1403, user nix-grpc-daemon (trusted) worker2 # [ 116.399321] nix-daemon[840]: reaped child process 1406, status = succeeded lb # [ 116.403745] niks3-server[1102]: 2026/09/23 04:13:11 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 116.419378] nix-grpc-daemon[1314]: time=2026-09-23T04:13:10.997Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 116.421072] nix-grpc-daemon[1314]: time=2026-09-23T04:13:10.997Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)" lb # [ 116.442761] niks3-server[1102]: 2026/09/23 04:13:11 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv lb # [ 116.461694] niks3-server[1102]: 2026/09/23 04:13:11 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst worker2 # [ 116.474250] nix-grpc-daemon[1314]: time=2026-09-23T04:13:11.053Z level=INFO msg="Uploading 1 narinfos" lb # [ 116.473525] niks3-server[1102]: 2026/09/23 04:13:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign lb # [ 116.480378] niks3-server[1102]: 2026/09/23 04:13:11 INFO Signed narinfos id=40 count=1 lb # [ 116.485404] niks3-server[1102]: 2026/09/23 04:13:11 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls lb # [ 116.494523] niks3-server[1102]: 2026/09/23 04:13:11 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete lb # [ 116.498546] niks3-server[1102]: 2026/09/23 04:13:11 INFO Completed upload id=40 worker2 # [ 116.501513] nix-grpc-daemon[1314]: time=2026-09-23T04:13:11.080Z level=INFO msg="Upload complete. (236ms)" lb # [ 116.502849] niks3-server[1102]: 2026/09/23 04:13:11 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo worker2 # [ 116.520448] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49964 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=1 worker2 # [ 116.536574] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:11Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=0 worker2 # [ 116.541279] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:11Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 116.545776] nix-daemon[840]: reaped child process 1385, status = succeeded worker2 # [ 116.563013] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49964 worker2 # [ 116.577381] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:49964 worker2 # [ 116.591421] nix-grpc-daemon[1314]: time=2026-09-23T04:13:11.170Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'... worker2 # [ 116.602305] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49964 worker2 # [ 116.616688] nix-daemon[840]: accepted connection from pid 1253, user nix-grpc-daemon (trusted) worker2 # [ 116.768113] nix-daemon[840]: accepted connection from pid 1412, user nix-grpc-daemon (trusted) worker2 # [ 116.787575] nix-daemon[840]: reaped child process 1415, status = succeeded lb # [ 116.788994] niks3-server[1102]: 2026/09/23 04:13:11 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 116.801340] nix-grpc-daemon[1314]: time=2026-09-23T04:13:11.380Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 116.802944] nix-grpc-daemon[1314]: time=2026-09-23T04:13:11.380Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)" lb # [ 116.818331] niks3-server[1102]: 2026/09/23 04:13:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign worker2 # [ 116.830909] nix-grpc-daemon[1314]: time=2026-09-23T04:13:11.409Z level=INFO msg="Uploading 1 narinfos" lb # [ 116.834122] niks3-server[1102]: 2026/09/23 04:13:11 INFO Signed narinfos id=41 count=1 lb # [ 116.841014] niks3-server[1102]: 2026/09/23 04:13:11 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls lb # [ 116.852889] niks3-server[1102]: 2026/09/23 04:13:11 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst lb # [ 116.859576] niks3-server[1102]: 2026/09/23 04:13:11 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete worker2 # [ 116.861190] nix-grpc-daemon[1314]: time=2026-09-23T04:13:11.440Z level=INFO msg="Upload complete. (229ms)" lb # [ 116.868261] niks3-server[1102]: 2026/09/23 04:13:11 INFO Completed upload id=41 worker2 # [ 116.870744] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49964 duration_s=0 paths=1 nar_bytes_in=784 lb # [ 116.870310] niks3-server[1102]: 2026/09/23 04:13:11 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo worker2 # [ 116.880701] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49964 worker2 # [ 116.891163] nix-daemon[840]: accepted connection from pid 1253, user nix-grpc-daemon (trusted) worker2 # [ 116.895701] nix-daemon[840]: reaped child process 1410, status = succeeded client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051... worker2 # [ 116.908228] nix-daemon[840]: accepted connection from pid 1253, user nix-grpc-daemon (trusted) client # farm-top> worker2: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-after-niks3-restart worker2 # [ 118.159518] nix-daemon[840]: reaped child process 1419, status = succeeded worker2 # [ 118.286759] nix-daemon[840]: accepted connection from pid 1435, user nix-grpc-daemon (trusted) worker2 # [ 118.306829] nix-daemon[840]: reaped child process 1438, status = succeeded lb # [ 118.308681] niks3-server[1102]: 2026/09/23 04:13:12 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 118.325521] nix-grpc-daemon[1314]: time=2026-09-23T04:13:12.903Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 118.327323] nix-grpc-daemon[1314]: time=2026-09-23T04:13:12.903Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)" lb # [ 118.358624] niks3-server[1102]: 2026/09/23 04:13:12 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst worker2 # [ 118.374024] nix-grpc-daemon[1314]: time=2026-09-23T04:13:12.952Z level=INFO msg="Uploading 1 narinfos" lb # [ 118.372343] niks3-server[1102]: 2026/09/23 04:13:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign lb # [ 118.379799] niks3-server[1102]: 2026/09/23 04:13:12 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv lb # [ 118.386340] niks3-server[1102]: 2026/09/23 04:13:12 INFO Signed narinfos id=42 count=1 lb # [ 118.393596] niks3-server[1102]: 2026/09/23 04:13:12 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls lb # [ 118.397671] niks3-server[1102]: 2026/09/23 04:13:13 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete worker2 # [ 118.400186] nix-grpc-daemon[1314]: time=2026-09-23T04:13:12.979Z level=INFO msg="Upload complete. (280ms)" lb # [ 118.401063] niks3-server[1102]: 2026/09/23 04:13:13 INFO Completed upload id=42 lb # [ 118.405382] niks3-server[1102]: 2026/09/23 04:13:13 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo worker2 # [ 118.413129] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49964 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=1 worker2 # [ 118.433549] nix-daemon[840]: reaped child process 1417, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2, in 6.38 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath, in 0.17 seconds) (finished: subtest: a niks3 restart does not cost the leader its role, in 7.04 seconds) subtest: scheduler down: the other node takes the lock and keeps it lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2 # [ 118.879557] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 119.702664] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:14Z level=info event=signal_exit lb # [ 119.722132] niks3-server[1102]: 2026/09/23 04:13:14 INFO lead: released remote=192.168.1.4:56566 lb # [ 119.901083] envoy[906]: conn peer=[2001:db8:1::1]:60410 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1006 tx=304 ms=1599 lb # [ 119.906014] envoy[906]: conn peer=[2001:db8:1::1]:35226 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9749 tx=4449 ms=6119 worker1 # [ 120.721401] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:15Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 120.729997] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:15Z level=info event=scheduler_restarting worker2 # [ 120.733922] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:15Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 120.738725] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:15Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:49954 worker2 # [ 120.743764] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:15Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 120.749585] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:15Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:49954 worker2 # [ 120.760474] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:15Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 120.768900] nix-grpc-daemon[1253]: E0000 00:00:1790136795.347534 1299 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true worker2 # [ 120.782700] nix-grpc-daemon[1253]: E0000 00:00:1790136795.347717 1299 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true worker2 # [ 120.790505] nix-daemon[840]: reaped child process 1306, status = succeeded worker1 # [ 120.920930] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 120.926302] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 120.945189] nix-grpc-daemon[1253]: ts=2026-09-23T04:13:15Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker1 # [ 121.127078] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 121.132326] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 121.332981] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 121.337994] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 121.538452] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 121.543504] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 121.745854] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 121.749975] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 121.946287] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 121.951438] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 121.993559] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 122.000572] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 122.003151] systemd[1]: nix-grpc-daemon.service: Consumed 1.373s CPU time over 16.451s wall clock time, 39M memory peak, 47.3K incoming IP traffic, 83.1K outgoing IP traffic. worker2 # [ 122.018384] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over worker2 # [ 122.030134] systemd[1]: Closed Nix worker-protocol over gRPC. worker2 # [ 122.031691] systemd[1]: nix-grpc-daemon.socket: Consumed 867us CPU time over 17.336s wall clock time, 256K memory peak, 38.6K incoming IP traffic, 29.3K outgoing IP traffic. worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker1 # [ 122.151929] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 122.155307] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 122.356119] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 122.361483] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 122.562051] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 122.567264] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 122.705547] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:17Z level=info event=scheduler_take_over lb # [ 122.707429] niks3-server[1102]: 2026/09/23 04:13:17 INFO lead: acquired remote=192.168.1.3:49812 worker1 # [ 122.769769] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 122.773961] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 122.970460] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 122.975939] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1 # [ 123.177284] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 123.188482] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) worker1 # [ 123.384620] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 123.393969] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:17Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:52686 worker1 # [ 123.399253] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:17Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2 worker1 # [ 124.517409] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 124.530674] nix-daemon[849]: accepted connection from pid 1223, user nix-grpc-daemon (trusted) client # don't know how to build these paths: client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker1 # [ 124.555201] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 124.571420] nix-grpc-daemon[1223]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 47 ms (attempt 1/5) worker1 # [ 124.627735] nix-grpc-daemon[1223]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 115 ms (attempt 2/5) worker1 # [ 124.744573] nix-grpc-daemon[1223]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 120 ms (attempt 3/5) worker1 # [ 124.866459] nix-grpc-daemon[1223]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 270 ms (attempt 4/5) worker1 # [ 125.147773] nix-grpc-daemon[1223]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 125.153884] nix-grpc-daemon[1223]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 125.184240] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:19Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=0 worker1 # [ 125.190127] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:19Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 125.235310] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 125.251027] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52698 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'... worker1 # [ 125.265245] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 125.279294] nix-daemon[849]: accepted connection from pid 1223, user nix-grpc-daemon (trusted) worker1 # [ 125.306064] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:19Z level=info event=niks3_push_started pid=1303 worker1 # [ 125.456547] nix-grpc-daemon[1303]: time=2026-09-23T04:13:20.058Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 125.561831] nix-daemon[849]: accepted connection from pid 1316, user nix-grpc-daemon (trusted) worker1 # [ 125.581117] nix-daemon[849]: reaped child process 1319, status = succeeded lb # [ 125.583456] niks3-server[1102]: 2026/09/23 04:13:20 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 125.592136] nix-grpc-daemon[1303]: time=2026-09-23T04:13:20.194Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 125.593590] nix-grpc-daemon[1303]: time=2026-09-23T04:13:20.194Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)" lb # [ 125.629659] niks3-server[1102]: 2026/09/23 04:13:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign worker1 # [ 125.642826] nix-grpc-daemon[1303]: time=2026-09-23T04:13:20.244Z level=INFO msg="Uploading 1 narinfos" lb # [ 125.643177] niks3-server[1102]: 2026/09/23 04:13:20 INFO Signed narinfos id=43 count=1 lb # [ 125.652790] niks3-server[1102]: 2026/09/23 04:13:20 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst lb # [ 125.663094] niks3-server[1102]: 2026/09/23 04:13:20 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls worker1 # [ 125.680087] nix-grpc-daemon[1303]: time=2026-09-23T04:13:20.282Z level=INFO msg="Upload complete. (219ms)" lb # [ 125.681353] niks3-server[1102]: 2026/09/23 04:13:20 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete lb # [ 125.691693] niks3-server[1102]: 2026/09/23 04:13:20 INFO Completed upload id=43 worker1 # [ 125.690336] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52698 duration_s=0 paths=1 nar_bytes_in=520 lb # [ 125.696374] niks3-server[1102]: 2026/09/23 04:13:20 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo worker1 # [ 125.704432] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 125.712666] nix-daemon[849]: accepted connection from pid 1223, user nix-grpc-daemon (trusted) worker1 # [ 125.716200] nix-daemon[849]: reaped child process 1301, status = succeeded client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051... client # farm-a> node-a: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker1 # [ 125.729026] nix-daemon[849]: accepted connection from pid 1223, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-on-standby worker1 # [ 127.437486] nix-daemon[849]: reaped child process 1323, status = succeeded worker1 # [ 127.515826] nix-daemon[849]: accepted connection from pid 1339, user nix-grpc-daemon (trusted) worker1 # [ 127.531496] nix-daemon[849]: reaped child process 1342, status = succeeded lb # [ 127.540749] niks3-server[1102]: 2026/09/23 04:13:22 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 127.551086] nix-grpc-daemon[1303]: time=2026-09-23T04:13:22.151Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 127.553886] nix-grpc-daemon[1303]: time=2026-09-23T04:13:22.151Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)" lb # [ 127.580477] niks3-server[1102]: 2026/09/23 04:13:22 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst lb # [ 127.592778] niks3-server[1102]: 2026/09/23 04:13:22 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker1 # [ 127.601253] nix-grpc-daemon[1303]: time=2026-09-23T04:13:22.203Z level=INFO msg="Uploading 1 narinfos" lb # [ 127.605224] niks3-server[1102]: 2026/09/23 04:13:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign lb # [ 127.612694] niks3-server[1102]: 2026/09/23 04:13:22 INFO Signed narinfos id=44 count=1 lb # [ 127.614698] niks3-server[1102]: 2026/09/23 04:13:22 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls lb # [ 127.624592] niks3-server[1102]: 2026/09/23 04:13:22 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete lb # [ 127.629355] niks3-server[1102]: 2026/09/23 04:13:22 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo lb # [ 127.634023] niks3-server[1102]: 2026/09/23 04:13:22 INFO Completed upload id=44 worker1 # [ 127.631784] nix-grpc-daemon[1303]: time=2026-09-23T04:13:22.234Z level=INFO msg="Upload complete. (291ms)" worker1 # [ 127.653113] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52698 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=1 worker1 # [ 127.667970] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:22Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=0 worker1 # [ 127.671595] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:22Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 127.675863] nix-daemon[849]: reaped child process 1321, status = succeeded worker1 # [ 127.700653] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 127.711559] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 127.724914] nix-grpc-daemon[1303]: time=2026-09-23T04:13:22.327Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'... worker1 # [ 127.735293] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 127.748867] nix-daemon[849]: accepted connection from pid 1223, user nix-grpc-daemon (trusted) worker1 # [ 127.894995] nix-daemon[849]: accepted connection from pid 1348, user nix-grpc-daemon (trusted) worker1 # [ 127.913838] nix-daemon[849]: reaped child process 1351, status = succeeded lb # [ 127.913884] niks3-server[1102]: 2026/09/23 04:13:22 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 127.921642] nix-grpc-daemon[1303]: time=2026-09-23T04:13:22.523Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 127.923182] nix-grpc-daemon[1303]: time=2026-09-23T04:13:22.523Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)" worker1 # [ 127.951220] nix-grpc-daemon[1303]: time=2026-09-23T04:13:22.553Z level=INFO msg="Uploading 1 narinfos" lb # [ 127.946257] niks3-server[1102]: 2026/09/23 04:13:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign lb # [ 127.960486] niks3-server[1102]: 2026/09/23 04:13:22 INFO Signed narinfos id=45 count=1 lb # [ 127.963622] niks3-server[1102]: 2026/09/23 04:13:22 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls worker1 # [ 127.979612] nix-grpc-daemon[1303]: time=2026-09-23T04:13:22.582Z level=INFO msg="Upload complete. (215ms)" lb # [ 127.980464] niks3-server[1102]: 2026/09/23 04:13:22 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst lb # [ 127.987214] niks3-server[1102]: 2026/09/23 04:13:22 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete worker1 # [ 127.989959] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52698 duration_s=0 paths=1 nar_bytes_in=640 lb # [ 127.995274] niks3-server[1102]: 2026/09/23 04:13:22 INFO Completed upload id=45 lb # [ 127.997866] niks3-server[1102]: 2026/09/23 04:13:22 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo worker1 # [ 128.003398] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 128.012318] nix-daemon[849]: accepted connection from pid 1223, user nix-grpc-daemon (trusted) worker1 # [ 128.015527] nix-daemon[849]: reaped child process 1346, status = succeeded client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051... client # farm-b> node-a: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker1 # [ 128.028483] nix-daemon[849]: accepted connection from pid 1223, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-on-standby worker1 # [ 129.100740] nix-daemon[849]: reaped child process 1355, status = succeeded worker1 # [ 129.211256] nix-daemon[849]: accepted connection from pid 1371, user nix-grpc-daemon (trusted) worker1 # [ 129.227448] nix-daemon[849]: reaped child process 1374, status = succeeded lb # [ 129.236744] niks3-server[1102]: 2026/09/23 04:13:23 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 129.246943] nix-grpc-daemon[1303]: time=2026-09-23T04:13:23.848Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 129.248668] nix-grpc-daemon[1303]: time=2026-09-23T04:13:23.848Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)" lb # [ 129.275286] niks3-server[1102]: 2026/09/23 04:13:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign worker1 # [ 129.288245] nix-grpc-daemon[1303]: time=2026-09-23T04:13:23.890Z level=INFO msg="Uploading 1 narinfos" lb # [ 129.293838] niks3-server[1102]: 2026/09/23 04:13:23 INFO Signed narinfos id=46 count=1 lb # [ 129.302658] niks3-server[1102]: 2026/09/23 04:13:23 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv lb # [ 129.312242] niks3-server[1102]: 2026/09/23 04:13:23 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst worker1 # [ 129.322741] nix-grpc-daemon[1303]: time=2026-09-23T04:13:23.925Z level=INFO msg="Upload complete. (283ms)" lb # [ 129.324416] niks3-server[1102]: 2026/09/23 04:13:23 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls lb # [ 129.327722] niks3-server[1102]: 2026/09/23 04:13:23 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete lb # [ 129.334518] niks3-server[1102]: 2026/09/23 04:13:23 INFO Completed upload id=46 lb # [ 129.337376] niks3-server[1102]: 2026/09/23 04:13:23 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo worker1 # [ 129.340703] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52698 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=1 worker1 # [ 129.355828] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:23Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=0 worker1 # [ 129.360625] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:23Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 129.365246] nix-daemon[849]: reaped child process 1353, status = succeeded worker1 # [ 129.395221] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 129.410364] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 129.424271] nix-grpc-daemon[1303]: time=2026-09-23T04:13:24.026Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'... worker1 # [ 129.435608] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 129.450282] nix-daemon[849]: accepted connection from pid 1223, user nix-grpc-daemon (trusted) worker1 # [ 129.603126] nix-daemon[849]: accepted connection from pid 1380, user nix-grpc-daemon (trusted) worker1 # [ 129.619814] nix-daemon[849]: reaped child process 1383, status = succeeded lb # [ 129.626200] niks3-server[1102]: 2026/09/23 04:13:24 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 129.633969] nix-grpc-daemon[1303]: time=2026-09-23T04:13:24.236Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 129.635495] nix-grpc-daemon[1303]: time=2026-09-23T04:13:24.236Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)" worker1 # [ 129.654682] nix-grpc-daemon[1303]: time=2026-09-23T04:13:24.257Z level=INFO msg="Uploading 1 narinfos" lb # [ 129.655505] niks3-server[1102]: 2026/09/23 04:13:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign lb # [ 129.668783] niks3-server[1102]: 2026/09/23 04:13:24 INFO Signed narinfos id=47 count=1 lb # [ 129.677191] niks3-server[1102]: 2026/09/23 04:13:24 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst lb # [ 129.691531] niks3-server[1102]: 2026/09/23 04:13:24 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls worker1 # [ 129.696338] nix-grpc-daemon[1303]: time=2026-09-23T04:13:24.298Z level=INFO msg="Upload complete. (231ms)" lb # [ 129.697902] niks3-server[1102]: 2026/09/23 04:13:24 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete lb # [ 129.704968] niks3-server[1102]: 2026/09/23 04:13:24 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo lb # [ 129.710404] niks3-server[1102]: 2026/09/23 04:13:24 INFO Completed upload id=47 worker1 # [ 129.708138] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52698 duration_s=0 paths=1 nar_bytes_in=768 worker1 # [ 129.720476] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 129.731195] nix-daemon[849]: accepted connection from pid 1223, user nix-grpc-daemon (trusted) worker1 # [ 129.734709] nix-daemon[849]: reaped child process 1378, status = succeeded client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051... worker1 # [ 129.746186] nix-daemon[849]: accepted connection from pid 1223, user nix-grpc-daemon (trusted) client # farm-top> node-a: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv lb # [ 129.899179] envoy[906]: conn peer=[2001:db8:1::4]:58660 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=5513 tx=665 ms=15188 client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-on-standby worker1 # [ 131.424941] nix-daemon[849]: reaped child process 1387, status = succeeded worker1 # [ 131.527364] nix-daemon[849]: accepted connection from pid 1403, user nix-grpc-daemon (trusted) worker1 # [ 131.544400] nix-daemon[849]: reaped child process 1406, status = succeeded lb # [ 131.551198] niks3-server[1102]: 2026/09/23 04:13:26 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 131.562931] nix-grpc-daemon[1303]: time=2026-09-23T04:13:26.164Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 131.565500] nix-grpc-daemon[1303]: time=2026-09-23T04:13:26.164Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)" lb # [ 131.594427] niks3-server[1102]: 2026/09/23 04:13:26 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls worker1 # [ 131.602685] nix-grpc-daemon[1303]: time=2026-09-23T04:13:26.204Z level=INFO msg="Uploading 1 narinfos" lb # [ 131.604901] niks3-server[1102]: 2026/09/23 04:13:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign lb # [ 131.615561] niks3-server[1102]: 2026/09/23 04:13:26 INFO Signed narinfos id=48 count=1 lb # [ 131.620423] niks3-server[1102]: 2026/09/23 04:13:26 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst lb # [ 131.623726] niks3-server[1102]: 2026/09/23 04:13:26 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv lb # [ 131.634511] niks3-server[1102]: 2026/09/23 04:13:26 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete lb # [ 131.638240] niks3-server[1102]: 2026/09/23 04:13:26 INFO Completed upload id=48 lb # [ 131.640906] niks3-server[1102]: 2026/09/23 04:13:26 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo worker1 # [ 131.632770] nix-grpc-daemon[1303]: time=2026-09-23T04:13:26.235Z level=INFO msg="Upload complete. (272ms)" worker1 # [ 131.645908] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52698 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=1 worker1 # [ 131.664806] nix-daemon[849]: reaped child process 1385, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 7.40 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.18 seconds) worker2 # [ 131.955374] systemd[1]: Listening on Nix worker-protocol over gRPC. worker2 # [ 131.964355] systemd[1]: Starting scheduler-token.service... worker2 # [ 132.043891] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 132.052205] systemd[1]: Finished scheduler-token.service. worker2 # [ 132.053727] systemd[1]: scheduler-token.service: Consumed 46ms CPU time over 81ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 132.059291] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 132.181907] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:26Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 132.184318] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:26Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 132.188429] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:26Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 132.195226] nix-grpc-daemon[1459]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 132.212427] nix-grpc-daemon[1459]: E0000 00:00:1790136806.772153 1459 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 132.244273] nix-grpc-daemon[1459]: E0000 00:00:1790136806.774372 1459 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 132.252831] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:26Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2 # [ 132.255125] nix-grpc-daemon[1459]: E0000 00:00:1790136806.797692 1500 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 132.260518] nix-grpc-daemon[1459]: E0000 00:00:1790136806.812933 1500 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 132.264896] nix-grpc-daemon[1459]: E0000 00:00:1790136806.813067 1500 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 132.267722] nix-grpc-daemon[1459]: E0000 00:00:1790136806.831231 1500 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 132.270676] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 132.271492] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:26Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 132.269973] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:26Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 132.278879] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:26Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:52686 worker1 # [ 132.290397] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:26Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: scheduler down: the other node takes the lock and keeps it, in 13.87 seconds) subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs client # [ 132.561241] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs. client # Running as unit: rs.service; invocation ID: 638ba5feb21a47e097990df0096f5f6b client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.09 seconds) worker1 # [ 132.755819] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:52698 client # [ 132.768490] nix[1402]: don't know how to build these paths: client # [ 132.769481] nix[1402]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker1 # [ 132.771532] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:27Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=0 worker1 # [ 132.778369] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:27Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 132.818833] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 132.834592] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52698 client # [ 132.852237] nix[1402]: copying 1 paths to 'lb:50051'... worker1 # [ 132.848780] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52698 client # [ 132.856683] nix[1402]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'... worker1 # [ 132.863762] nix-daemon[849]: accepted connection from pid 1223, user nix-grpc-daemon (trusted) worker1 # [ 133.019237] nix-daemon[849]: accepted connection from pid 1419, user nix-grpc-daemon (trusted) worker1 # [ 133.039188] nix-daemon[849]: reaped child process 1422, status = succeeded lb # [ 133.044828] niks3-server[1102]: 2026/09/23 04:13:27 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 133.052910] nix-grpc-daemon[1303]: time=2026-09-23T04:13:27.655Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 133.054992] nix-grpc-daemon[1303]: time=2026-09-23T04:13:27.657Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)" lb # [ 133.078223] niks3-server[1102]: 2026/09/23 04:13:27 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls worker1 # [ 133.086895] nix-grpc-daemon[1303]: time=2026-09-23T04:13:27.689Z level=INFO msg="Uploading 1 narinfos" lb # [ 133.091004] niks3-server[1102]: 2026/09/23 04:13:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign lb # [ 133.101324] niks3-server[1102]: 2026/09/23 04:13:27 INFO Signed narinfos id=49 count=1 lb # [ 133.107929] niks3-server[1102]: 2026/09/23 04:13:27 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst lb # [ 133.116144] niks3-server[1102]: 2026/09/23 04:13:27 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete lb # [ 133.123990] niks3-server[1102]: 2026/09/23 04:13:27 INFO Completed upload id=49 worker1 # [ 133.122065] nix-grpc-daemon[1303]: time=2026-09-23T04:13:27.724Z level=INFO msg="Upload complete. (242ms)" lb # [ 133.126347] niks3-server[1102]: 2026/09/23 04:13:27 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo worker1 # [ 133.132979] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52698 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 133.152885] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52698 worker1 # [ 133.156879] nix-daemon[849]: accepted connection from pid 1223, user nix-grpc-daemon (trusted) worker1 # [ 133.162879] nix-daemon[849]: reaped child process 1417, status = succeeded client # [ 133.177600] nix[1402]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051... worker1 # [ 133.174222] nix-daemon[849]: accepted connection from pid 1223, user nix-grpc-daemon (trusted) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true worker2 # [ 135.369101] systemd-journald[271]: Received client request to rotate journal, rotating. worker2 # [ 135.389832] systemd-journald[271]: Vacuuming done, freed 0B of archived journals from /var/log/journal/98a9a6e0b8854ff2a37e9e979230b133. worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.09 seconds) worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service worker1 # [ 135.444360] systemd[1]: Stopping Nix worker-protocol over gRPC... worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds) client: must succeed: sleep 5.0 worker1 # [ 136.433122] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:31Z level=info event=draining lb # [ 139.900576] envoy[906]: conn peer=[2001:db8:1::1]:45396 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9649 tx=4457 ms=7147 client: (finished: must succeed: sleep 5.0, in 5.05 seconds) worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=' worker1 # [ 140.812557] nix-daemon[849]: reaped child process 1426, status = succeeded worker1 # [ 140.919380] nix-daemon[849]: accepted connection from pid 1467, user nix-grpc-daemon (trusted) worker1 # [ 140.936463] nix-daemon[849]: reaped child process 1470, status = succeeded lb # [ 140.945925] niks3-server[1102]: 2026/09/23 04:13:35 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 140.956407] nix-grpc-daemon[1303]: time=2026-09-23T04:13:35.556Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 140.957906] nix-grpc-daemon[1303]: time=2026-09-23T04:13:35.556Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)" lb # [ 140.985391] niks3-server[1102]: 2026/09/23 04:13:35 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst worker1 # [ 141.000461] nix-grpc-daemon[1303]: time=2026-09-23T04:13:35.602Z level=INFO msg="Uploading 1 narinfos" lb # [ 141.004486] niks3-server[1102]: 2026/09/23 04:13:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign lb # [ 141.015823] niks3-server[1102]: 2026/09/23 04:13:35 INFO Signed narinfos id=50 count=1 lb # [ 141.025050] niks3-server[1102]: 2026/09/23 04:13:35 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls worker1 # [ 141.023833] nix-grpc-daemon[1303]: time=2026-09-23T04:13:35.626Z level=INFO msg="Upload complete. (231ms)" lb # [ 141.027668] niks3-server[1102]: 2026/09/23 04:13:35 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete lb # [ 141.040973] niks3-server[1102]: 2026/09/23 04:13:35 INFO Completed upload id=50 worker1 # [ 141.034771] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52698 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=1 lb # [ 141.042609] niks3-server[1102]: 2026/09/23 04:13:35 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo worker1 # [ 141.053039] nix-daemon[849]: reaped child process 1424, status = succeeded client # [ 141.084774] systemd[1]: rs.service: Deactivated successfully. client # [ 141.089045] systemd[1]: rs.service: Consumed 192ms CPU time over 8.524s wall clock time, 41.2M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic. worker1 # [ 141.437375] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:36Z level=info event=signal_exit lb # [ 141.719840] niks3-server[1102]: 2026/09/23 04:13:36 INFO lead: released remote=192.168.1.3:49812 lb # [ 142.211190] niks3-server[1102]: 2026/09/23 04:13:36 INFO lead: acquired remote=192.168.1.4:56874 worker2 # [ 142.214318] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:36Z level=info event=scheduler_take_over worker2 # [ 142.716819] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:37Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 142.717990] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:37Z level=info event=scheduler_restarting worker1 # [ 142.722606] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:37Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 142.727703] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:37Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:52686 worker1 # [ 142.732594] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:37Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 142.737237] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:37Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:52686 worker1 # [ 142.743076] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:37Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 142.765897] nix-daemon[849]: reaped child process 1293, status = succeeded worker2 # [ 142.918256] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 142.933714] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:37Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 142.936915] nix-grpc-daemon[1223]: ts=2026-09-23T04:13:37Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 142.941480] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:37Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:52084 worker2 # [ 142.951566] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:37Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.20 seconds) client: waiting for success: ! systemctl is-active rs client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds) client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; } client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds) client: must fail: journalctl -u rs | grep -q 'reconnecting' client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker1 # [ 143.264103] nix-grpc-daemon[1223]: E0000 00:00:1790136817.866426 1269 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true worker1 # [ 143.271656] nix-grpc-daemon[1223]: E0000 00:00:1790136817.873914 1269 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker1 # [ 144.373509] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 144.379906] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 144.384498] systemd[1]: nix-grpc-daemon.service: Consumed 1.591s CPU time over 34.132s wall clock time, 38.8M memory peak, 49K incoming IP traffic, 92.8K outgoing IP traffic. worker1 # [ 144.403320] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 144.539719] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:39Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 144.542142] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:39Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 144.545974] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:39Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 144.551378] nix-grpc-daemon[1477]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 144.568159] nix-grpc-daemon[1477]: E0000 00:00:1790136819.153236 1477 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 144.570604] nix-grpc-daemon[1477]: E0000 00:00:1790136819.156909 1477 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 144.596433] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:39Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 144.598668] nix-grpc-daemon[1477]: E0000 00:00:1790136819.163081 1520 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 144.601365] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 144.602210] nix-grpc-daemon[1477]: E0000 00:00:1790136819.183484 1520 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 144.607526] nix-grpc-daemon[1477]: E0000 00:00:1790136819.183655 1520 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 144.612194] nix-grpc-daemon[1477]: E0000 00:00:1790136819.198017 1520 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 144.614675] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 144.618668] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:39Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:52084 worker2 # [ 144.624146] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:39Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.98 seconds) subtest: scheduler restart mid-build: build finishes, no double build client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid client # [ 145.539540] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid. client # Running as unit: mid.service; invocation ID: a81327ed3ff74f1ca390b124539c97cc client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.09 seconds) worker2 # [ 145.748784] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:52088 worker2 # [ 145.766720] nix-daemon[840]: accepted connection from pid 1459, user nix-grpc-daemon (trusted) client # [ 145.781057] nix[1438]: don't know how to build these paths: client # [ 145.782698] nix[1438]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker2 # [ 145.793165] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:40Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker2 # [ 145.799338] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:40Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 145.847439] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52088 worker2 # [ 145.863135] nix-grpc-daemon[1459]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 17 ms (attempt 1/5) worker2 # [ 145.889241] nix-grpc-daemon[1459]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 50 ms (attempt 2/5) worker2 # [ 145.940917] nix-grpc-daemon[1459]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 50 ms (attempt 3/5) worker2 # [ 145.992716] nix-grpc-daemon[1459]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 566 ms (attempt 4/5) worker2 # [ 146.568354] nix-grpc-daemon[1459]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 146.574664] nix-grpc-daemon[1459]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 146.593432] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52088 client # [ 146.607006] nix[1438]: copying 1 paths to 'lb:50051'... worker2 # [ 146.611966] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52088 worker2 # [ 146.626488] nix-daemon[840]: accepted connection from pid 1459, user nix-grpc-daemon (trusted) worker2 # [ 146.659787] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:41Z level=info event=niks3_push_started pid=1585 worker2 # [ 146.805894] nix-grpc-daemon[1585]: time=2026-09-23T04:13:41.381Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 146.928298] nix-daemon[840]: accepted connection from pid 1601, user nix-grpc-daemon (trusted) worker2 # [ 146.945514] nix-daemon[840]: reaped child process 1604, status = succeeded lb # [ 146.949849] niks3-server[1102]: 2026/09/23 04:13:41 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 146.966982] nix-grpc-daemon[1585]: time=2026-09-23T04:13:41.545Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 146.970873] nix-grpc-daemon[1585]: time=2026-09-23T04:13:41.545Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)" lb # [ 147.002448] niks3-server[1102]: 2026/09/23 04:13:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign worker2 # [ 147.014696] nix-grpc-daemon[1585]: time=2026-09-23T04:13:41.592Z level=INFO msg="Uploading 1 narinfos" lb # [ 147.013112] niks3-server[1102]: 2026/09/23 04:13:41 INFO Signed narinfos id=51 count=1 lb # [ 147.020788] niks3-server[1102]: 2026/09/23 04:13:41 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls lb # [ 147.027996] niks3-server[1102]: 2026/09/23 04:13:41 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst lb # [ 147.037274] niks3-server[1102]: 2026/09/23 04:13:41 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete worker2 # [ 147.048759] nix-grpc-daemon[1585]: time=2026-09-23T04:13:41.627Z level=INFO msg="Upload complete. (245ms)" lb # [ 147.046404] niks3-server[1102]: 2026/09/23 04:13:41 INFO Completed upload id=51 lb # [ 147.056569] niks3-server[1102]: 2026/09/23 04:13:41 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo worker2 # [ 147.060155] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52088 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 147.074099] nix-grpc-daemon[1459]: ts=2026-09-23T04:13:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52088 worker2 # [ 147.082578] nix-daemon[840]: accepted connection from pid 1459, user nix-grpc-daemon (trusted) worker2 # [ 147.084769] nix-daemon[840]: reaped child process 1583, status = succeeded worker2 # [ 147.097435] nix-daemon[840]: accepted connection from pid 1459, user nix-grpc-daemon (trusted) client # [ 146.611775] nix[1438]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'... client # [ 147.096720] nix[1438]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service worker2 # [ 149.406751] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1459 (nix-grpc-daemon) on client request. worker2 # [ 149.408831] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request. lb # [ 149.426861] niks3-server[1102]: 2026/09/23 04:13:44 INFO lead: released remote=192.168.1.4:56874 worker2 # [ 149.435706] systemd[1]: Starting scheduler-token.service... client # [ 149.431560] nix[1438]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again worker1 # [ 149.426362] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination" client # [ 149.437995] nix[1438]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting worker2 # [ 149.448663] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL worker2 # [ 149.450140] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'. worker2 # [ 149.451416] systemd[1]: nix-grpc-daemon.service: Consumed 489ms CPU time over 17.376s wall clock time, 35M memory peak, 15.5K incoming IP traffic, 27.5K outgoing IP traffic. worker2 # [ 149.487918] nix-daemon[840]: reaped child process 1575, status = succeeded worker2 # [ 149.490057] nix-daemon[840]: reaped child process 1606, status = succeeded worker2 # [ 149.496413] nix-daemon[1608]: unexpected Nix daemon error: error: interrupted by the user worker2 # [ 149.507265] nix-daemon[840]: reaped child process 1608, status = failed with exit code 1 worker2 # [ 149.525291] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 149.526265] systemd[1]: Finished scheduler-token.service. worker2 # [ 149.527021] systemd[1]: scheduler-token.service: Consumed 24ms CPU time over 89ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 149.532708] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1. worker2 # [ 149.534197] systemd[1]: Starting scheduler-token.service... lb # [ 149.563182] niks3-server[1102]: 2026/09/23 04:13:44 INFO lead: acquired remote=192.168.1.3:38954 worker1 # [ 149.566869] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:44Z level=info event=scheduler_take_over worker2 # [ 149.580792] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 149.584136] systemd[1]: Finished scheduler-token.service. worker2 # [ 149.584926] systemd[1]: scheduler-token.service: Consumed 21ms CPU time over 50ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 149.588876] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 149.626337] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 149.690641] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:44Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 149.692867] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:44Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 149.696641] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:44Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 149.701895] nix-grpc-daemon[1651]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 149.703349] nix-grpc-daemon[1651]: E0000 00:00:1790136824.280388 1651 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.34 seconds) client: must succeed: sleep 5.0 worker1 # [ 149.721079] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker2 # [ 149.717832] nix-grpc-daemon[1651]: E0000 00:00:1790136824.286280 1651 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 149.746307] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:44Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 149.747978] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 149.754882] nix-grpc-daemon[1651]: E0000 00:00:1790136824.297462 1691 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 149.757811] nix-grpc-daemon[1651]: E0000 00:00:1790136824.314153 1691 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 149.760301] nix-grpc-daemon[1651]: E0000 00:00:1790136824.314288 1691 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 149.765370] nix-grpc-daemon[1651]: E0000 00:00:1790136824.333484 1691 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 149.769889] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 149.771763] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" lb # [ 149.902840] envoy[906]: conn peer=[2001:db8:1::1]:39736 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=1516 ms=8306 lb # [ 149.908049] envoy[906]: conn peer=[2001:db8:1::3]:45056 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=3126 tx=1050 ms=32447 lb # [ 149.912972] envoy[906]: conn peer=[2001:db8:1::4]:48748 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=1935 tx=315 ms=17174 worker1 # [ 149.921885] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 149.927292] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" client # [ 149.936086] nix[1438]: scheduler at lb:50051: scheduler passive, another node serves, reconnecting worker2 # [ 149.973029] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 149.979496] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker1 # [ 150.128070] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 150.133277] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker2 # [ 150.180559] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 150.187127] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker1 # [ 150.334059] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:44Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 150.339420] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 150.388225] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 150.393180] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 150.541781] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 150.545926] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 150.594059] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 150.599502] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 150.742081] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 150.747342] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 150.802106] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 150.806341] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" client # [ 150.939634] nix[1438]: scheduler at lb:50051: no healthy upstream, reconnecting worker1 # [ 150.948073] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 150.953138] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 151.002733] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 151.007944] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 151.153892] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 151.159034] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 151.209259] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 151.214421] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 151.359767] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 151.365101] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 151.415391] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:45Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 151.433524] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:46Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 151.438180] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:46Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:50608 worker1 # [ 151.444626] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:46Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 151.568244] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:46Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 151.572346] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:46Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:50608 worker1 # [ 151.577134] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:46Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 152.945830] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:47Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker1 # [ 152.951847] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:47Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 153.001125] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56656 worker2 # [ 153.016425] nix-daemon[840]: accepted connection from pid 1651, user nix-grpc-daemon (trusted) worker2 # [ 153.039274] nix-daemon[840]: accepted connection from pid 1651, user nix-grpc-daemon (trusted) client # [ 153.056566] nix[1438]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... worker2 # [ 153.058651] nix-daemon[840]: accepted connection from pid 1651, user nix-grpc-daemon (trusted) client: (finished: must succeed: sleep 5.0, in 5.05 seconds) client: waiting for success: ! systemctl is-active mid worker2 # [ 155.074956] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:49Z level=info event=niks3_push_started pid=1733 worker2 # [ 155.108822] nix-daemon[840]: reaped child process 1708, status = succeeded worker2 # [ 155.207856] nix-grpc-daemon[1733]: time=2026-09-23T04:13:49.786Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 155.318226] nix-daemon[840]: accepted connection from pid 1746, user nix-grpc-daemon (trusted) worker2 # [ 155.337781] nix-daemon[840]: reaped child process 1749, status = succeeded lb # [ 155.334409] niks3-server[1102]: 2026/09/23 04:13:49 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 155.348162] nix-grpc-daemon[1733]: time=2026-09-23T04:13:49.927Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 155.353156] nix-grpc-daemon[1733]: time=2026-09-23T04:13:49.929Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)" lb # [ 155.393801] niks3-server[1102]: 2026/09/23 04:13:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign lb # [ 155.408456] niks3-server[1102]: 2026/09/23 04:13:50 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls lb # [ 155.421769] niks3-server[1102]: 2026/09/23 04:13:50 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst lb # [ 155.444485] niks3-server[1102]: 2026/09/23 04:13:50 INFO Signed narinfos id=52 count=1 worker2 # [ 155.448151] nix-grpc-daemon[1733]: time=2026-09-23T04:13:50.026Z level=INFO msg="Uploading 1 narinfos" lb # [ 155.455470] niks3-server[1102]: 2026/09/23 04:13:50 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv lb # [ 155.467568] niks3-server[1102]: 2026/09/23 04:13:50 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete lb # [ 155.475725] niks3-server[1102]: 2026/09/23 04:13:50 INFO Completed upload id=52 worker2 # [ 155.478390] nix-grpc-daemon[1733]: time=2026-09-23T04:13:50.057Z level=INFO msg="Upload complete. (266ms)" lb # [ 155.483370] niks3-server[1102]: 2026/09/23 04:13:50 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo worker2 # [ 155.492362] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56656 duration_s=2 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=1 worker2 # [ 155.509988] nix-daemon[840]: reaped child process 1706, status = succeeded client # [ 155.525995] systemd[1]: mid.service: Deactivated successfully. client # [ 155.531023] systemd[1]: mid.service: Consumed 202ms CPU time over 9.987s wall clock time, 41.4M memory peak, 9.2K incoming IP traffic, 11.3K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active mid, in 1.12 seconds) client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; } client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.06 seconds) (finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.72 seconds) subtest: interrupting the client stops the build on the worker client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr client # [ 156.232467] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr. client # Running as unit: intr.service; invocation ID: 02485d03a7244f3b9112e24cf96b0a0c client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.05 seconds) worker1 # [ 156.416434] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 156.429293] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) client # [ 156.451976] nix[1474]: don't know how to build these paths: client # [ 156.454460] nix[1474]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker1 # [ 156.459858] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:51Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=0 worker1 # [ 156.466659] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:51Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 156.511536] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 156.527237] nix-grpc-daemon[1477]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 19 ms (attempt 1/5) worker1 # [ 156.555621] nix-grpc-daemon[1477]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 62 ms (attempt 2/5) worker1 # [ 156.626817] nix-grpc-daemon[1477]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 36 ms (attempt 3/5) worker1 # [ 156.664505] nix-grpc-daemon[1477]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 769 ms (attempt 4/5) worker1 # [ 157.445622] nix-grpc-daemon[1477]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 157.449005] nix-grpc-daemon[1477]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 157.460092] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 157.476889] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 client # [ 157.481175] nix[1474]: copying 1 paths to 'lb:50051'... client # [ 157.486548] nix[1474]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'... worker1 # [ 157.488623] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 157.512333] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:52Z level=info event=niks3_push_started pid=1578 worker1 # [ 157.638482] nix-grpc-daemon[1578]: time=2026-09-23T04:13:52.240Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 157.745530] nix-daemon[849]: accepted connection from pid 1590, user nix-grpc-daemon (trusted) worker1 # [ 157.764688] nix-daemon[849]: reaped child process 1593, status = succeeded lb # [ 157.765622] niks3-server[1102]: 2026/09/23 04:13:52 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 157.778017] nix-grpc-daemon[1578]: time=2026-09-23T04:13:52.379Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 157.779509] nix-grpc-daemon[1578]: time=2026-09-23T04:13:52.379Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)" lb # [ 157.810883] niks3-server[1102]: 2026/09/23 04:13:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign worker1 # [ 157.822703] nix-grpc-daemon[1578]: time=2026-09-23T04:13:52.424Z level=INFO msg="Uploading 1 narinfos" lb # [ 157.823943] niks3-server[1102]: 2026/09/23 04:13:52 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls lb # [ 157.832257] niks3-server[1102]: 2026/09/23 04:13:52 INFO Signed narinfos id=53 count=1 lb # [ 157.834541] niks3-server[1102]: 2026/09/23 04:13:52 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst lb # [ 157.847455] niks3-server[1102]: 2026/09/23 04:13:52 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete lb # [ 157.853348] niks3-server[1102]: 2026/09/23 04:13:52 INFO Completed upload id=53 worker1 # [ 157.852423] nix-grpc-daemon[1578]: time=2026-09-23T04:13:52.454Z level=INFO msg="Upload complete. (210ms)" lb # [ 157.855769] niks3-server[1102]: 2026/09/23 04:13:52 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo worker1 # [ 157.860301] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=424 worker1 # [ 157.868335] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 157.877370] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 157.881856] nix-daemon[849]: reaped child process 1576, status = succeeded worker1 # [ 157.892373] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) client # [ 157.896662] nix[1474]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051... client: must succeed: systemctl kill -s INT intr worker1 # [ 159.898716] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:54Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv client # [ 159.901413] systemd[1]: intr.service: Sent signal SIGINT to main process 1474 (nix) on client request. lb # [ 159.899929] envoy[906]: conn peer=[2001:db8:1::1]:57108 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4314 tx=2093 ms=9770 client # [ 159.910804] nix[1474]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client # [ 159.922582] nix[1474]: error: interrupted by the user worker1 # [ 159.918759] nix-daemon[1597]: unexpected Nix daemon error: error: interrupted by the user client: (finished: must succeed: systemctl kill -s INT intr, in 0.08 seconds) worker1 # [ 159.928976] nix-daemon[849]: reaped child process 1595, status = succeeded worker1 # [ 159.930164] nix-daemon[849]: reaped child process 1597, status = failed with exit code 1 client # [ 159.949115] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE client # [ 159.954234] systemd[1]: intr.service: Failed with result 'exit-code'. client # [ 159.956675] systemd[1]: intr.service: Consumed 166ms CPU time over 3.720s wall clock time, 41.2M memory peak, 6.4K incoming IP traffic, 8.3K outgoing IP traffic. (finished: subtest: interrupting the client stops the build on the worker, in 3.84 seconds) subtest: a deploy mid-build drains: the build finishes, the new generation takes over client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw client # [ 160.093797] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw. client # Running as unit: sw.service; invocation ID: 29452575d82544269d6e270484846455 client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.08 seconds) client # [ 160.287682] nix[1495]: don't know how to build these paths: client # [ 160.289312] nix[1495]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker2 # [ 160.285980] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:56656 worker1 # [ 160.291424] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:54Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=0 worker1 # [ 160.297310] nix-grpc-daemon[1477]: ts=2026-09-23T04:13:54Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 160.342683] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56656 worker2 # [ 160.358421] nix-grpc-daemon[1651]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 7 ms (attempt 1/5) worker2 # [ 160.374551] nix-grpc-daemon[1651]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 60 ms (attempt 2/5) worker2 # [ 160.436496] nix-grpc-daemon[1651]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 219 ms (attempt 3/5) worker2 # [ 160.657592] nix-grpc-daemon[1651]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 135 ms (attempt 4/5) worker2 # [ 160.794853] nix-grpc-daemon[1651]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 160.801224] nix-grpc-daemon[1651]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 160.820068] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56656 client # [ 160.836065] nix[1495]: copying 1 paths to 'lb:50051'... worker2 # [ 160.834879] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56656 client # [ 160.838768] nix[1495]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'... worker2 # [ 160.850110] nix-daemon[840]: accepted connection from pid 1651, user nix-grpc-daemon (trusted) worker2 # [ 161.011036] nix-daemon[840]: accepted connection from pid 1781, user nix-grpc-daemon (trusted) worker2 # [ 161.030694] nix-daemon[840]: reaped child process 1784, status = succeeded lb # [ 161.031578] niks3-server[1102]: 2026/09/23 04:13:55 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 161.045363] nix-grpc-daemon[1733]: time=2026-09-23T04:13:55.623Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 161.046931] nix-grpc-daemon[1733]: time=2026-09-23T04:13:55.623Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)" lb # [ 161.062644] niks3-server[1102]: 2026/09/23 04:13:55 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst worker2 # [ 161.080091] nix-grpc-daemon[1733]: time=2026-09-23T04:13:55.657Z level=INFO msg="Uploading 1 narinfos" lb # [ 161.077336] niks3-server[1102]: 2026/09/23 04:13:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign lb # [ 161.087992] niks3-server[1102]: 2026/09/23 04:13:55 INFO Signed narinfos id=54 count=1 worker2 # [ 161.105431] nix-grpc-daemon[1733]: time=2026-09-23T04:13:55.684Z level=INFO msg="Upload complete. (238ms)" lb # [ 161.103329] niks3-server[1102]: 2026/09/23 04:13:55 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls lb # [ 161.110770] niks3-server[1102]: 2026/09/23 04:13:55 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete worker2 # [ 161.116347] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56656 duration_s=0 paths=1 nar_bytes_in=416 lb # [ 161.117724] niks3-server[1102]: 2026/09/23 04:13:55 INFO Completed upload id=54 lb # [ 161.120331] niks3-server[1102]: 2026/09/23 04:13:55 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo worker2 # [ 161.131963] nix-grpc-daemon[1651]: ts=2026-09-23T04:13:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56656 worker2 # [ 161.138706] nix-daemon[840]: accepted connection from pid 1651, user nix-grpc-daemon (trusted) worker2 # [ 161.143771] nix-daemon[840]: reaped child process 1779, status = succeeded worker2 # [ 161.155053] nix-daemon[840]: accepted connection from pid 1651, user nix-grpc-daemon (trusted) client # [ 161.155191] nix[1495]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051... worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2 worker2 # Checking switch inhibitors... done worker2 # [ 164.286416] nixos[1825]: switching to system configuration /nix/store/x8kjjmji2ck289pjncrbp4826lif72v0-nixos-system-worker2-test worker2 # activating the configuration... worker2 # [ 164.292453] systemd[1]: Stopped target Local File Systems. worker2 # [ 164.293289] systemd[1]: Stopped target Remote File Systems. worker2 # setting up /etc... worker2 # [ 164.865575] systemd[1]: Reload requested from client PID 1825 ('.switch-to-conf') (unit backdoor.service)... worker2 # [ 164.867488] systemd[1]: Reloading... worker2 # [ 169.775857] systemd[1]: Reloading finished in 4903 ms. worker2 # restarting sysinit-reactivation.target lb # [ 169.903924] envoy[906]: conn peer=[2001:db8:1::1]:57906 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3013 tx=851 ms=3527 worker2 # [ 169.912721] systemd[1]: Starting Re-setup tmpfiles on a system that is already running.... worker2 # [ 170.038938] systemd[1]: Finished Re-setup tmpfiles on a system that is already running.. worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service worker2 # [ 170.044299] systemd[1]: Reached target Reactivate sysinit units. worker2 # [ 170.045168] systemd[1]: Reloading D-Bus System Message Bus... worker2 # [ 170.046265] dbus-broker-launch[494]: Caught SIGHUP, trigger reload. worker2 # [ 170.061202] systemd[1]: Reloading Nix worker-protocol over gRPC... worker2 # [ 170.064948] dbus-broker-launch[494]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 170.068851] dbus-broker-launch[494]: NSS returned no entry for 'systemd-timesync' worker2 # [ 170.070146] dbus-broker-launch[494]: Invalid user-name in /nix/store/ng2ydfkv03gvmdws4f4mjf9wj6qj3kq6-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker2 # [ 170.100953] systemd[1]: Reloaded Nix worker-protocol over gRPC. worker2 # [ 170.113508] systemd[1]: Reloaded D-Bus System Message Bus. worker2 # [ 170.114354] systemd[1]: Reached target Local File Systems. worker2 # [ 170.115137] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker2 # [ 170.129989] systemd[1]: Starting Create SUID/SGID Wrappers... worker2 # [ 170.130850] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker2 # [ 170.140096] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker2 # [ 170.142205] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 170.143704] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 170.152151] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 170.153376] systemd[1]: Reached target Remote File Systems. worker2 # [ 170.156914] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 170.158984] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 170.464488] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 170.465549] systemd[1]: Finished Create SUID/SGID Wrappers. worker2 # [ 170.475361] systemd[1]: Starting linger-users.service... worker2 # [ 170.517912] systemd[1]: linger-users.service: Deactivated successfully. worker2 # [ 170.518888] systemd[1]: Finished linger-users.service. worker2 # [ 170.715125] nix-grpc-daemon[1651]: ts=2026-09-23T04:14:05Z level=info event=draining worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service worker2 # [ 170.776335] nixos[1825]: finished switching to system configuration /nix/store/x8kjjmji2ck289pjncrbp4826lif72v0-nixos-system-worker2-test worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 6.96 seconds) worker2: must succeed: pgrep -f 'read -t [9]0 x' worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2 client # don't know how to build these paths: client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker1 # [ 171.051715] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 171.059662] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 171.085769] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:05Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=0 worker1 # [ 171.092731] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:05Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 171.136959] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 171.151889] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 client # copying 1 paths to 'lb:50051'... worker1 # [ 171.167720] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'... worker1 # [ 171.190542] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 171.347969] nix-daemon[849]: accepted connection from pid 1652, user nix-grpc-daemon (trusted) worker1 # [ 171.366074] nix-daemon[849]: reaped child process 1655, status = succeeded lb # [ 171.375496] niks3-server[1102]: 2026/09/23 04:14:05 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 171.384604] nix-grpc-daemon[1578]: time=2026-09-23T04:14:05.986Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 171.387028] nix-grpc-daemon[1578]: time=2026-09-23T04:14:05.988Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)" lb # [ 171.410911] niks3-server[1102]: 2026/09/23 04:14:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign worker1 # [ 171.419771] nix-grpc-daemon[1578]: time=2026-09-23T04:14:06.021Z level=INFO msg="Uploading 1 narinfos" lb # [ 171.423312] niks3-server[1102]: 2026/09/23 04:14:06 INFO Signed narinfos id=55 count=1 lb # [ 171.432861] niks3-server[1102]: 2026/09/23 04:14:06 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls lb # [ 171.441909] niks3-server[1102]: 2026/09/23 04:14:06 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst lb # [ 171.453160] niks3-server[1102]: 2026/09/23 04:14:06 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete lb # [ 171.461458] niks3-server[1102]: 2026/09/23 04:14:06 INFO Completed upload id=55 worker1 # [ 171.453109] nix-grpc-daemon[1578]: time=2026-09-23T04:14:06.055Z level=INFO msg="Upload complete. (245ms)" worker1 # [ 171.464518] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=528 lb # [ 171.469525] niks3-server[1102]: 2026/09/23 04:14:06 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo worker1 # [ 171.480802] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 171.490906] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 171.497625] nix-daemon[849]: reaped child process 1650, status = succeeded client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051... worker1 # [ 171.509470] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) client # farm-a> node-a: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-during-drain worker1 # [ 172.345179] nix-daemon[849]: reaped child process 1659, status = succeeded worker1 # [ 172.458912] nix-daemon[849]: accepted connection from pid 1675, user nix-grpc-daemon (trusted) worker1 # [ 172.475934] nix-daemon[849]: reaped child process 1678, status = succeeded lb # [ 172.484981] niks3-server[1102]: 2026/09/23 04:14:07 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 172.493625] nix-grpc-daemon[1578]: time=2026-09-23T04:14:07.094Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 172.496347] nix-grpc-daemon[1578]: time=2026-09-23T04:14:07.094Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)" lb # [ 172.518259] niks3-server[1102]: 2026/09/23 04:14:07 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst lb # [ 172.536102] niks3-server[1102]: 2026/09/23 04:14:07 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv lb # [ 172.549890] niks3-server[1102]: 2026/09/23 04:14:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign worker1 # [ 172.553614] nix-grpc-daemon[1578]: time=2026-09-23T04:14:07.155Z level=INFO msg="Uploading 1 narinfos" lb # [ 172.557579] niks3-server[1102]: 2026/09/23 04:14:07 INFO Signed narinfos id=56 count=1 lb # [ 172.563072] niks3-server[1102]: 2026/09/23 04:14:07 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls lb # [ 172.570861] niks3-server[1102]: 2026/09/23 04:14:07 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete lb # [ 172.575856] niks3-server[1102]: 2026/09/23 04:14:07 INFO Completed upload id=56 lb # [ 172.580300] niks3-server[1102]: 2026/09/23 04:14:07 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo worker1 # [ 172.569811] nix-grpc-daemon[1578]: time=2026-09-23T04:14:07.172Z level=INFO msg="Upload complete. (247ms)" worker1 # [ 172.587442] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=1 worker1 # [ 172.601571] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:07Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=0 worker1 # [ 172.605229] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:07Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 172.609061] nix-daemon[849]: reaped child process 1657, status = succeeded worker1 # [ 172.632601] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 172.642795] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 172.658294] nix-grpc-daemon[1578]: time=2026-09-23T04:14:07.260Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker1 # [ 172.669165] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'... worker1 # [ 172.690564] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 172.845889] nix-daemon[849]: accepted connection from pid 1684, user nix-grpc-daemon (trusted) worker1 # [ 172.866463] nix-daemon[849]: reaped child process 1687, status = succeeded lb # [ 172.868609] niks3-server[1102]: 2026/09/23 04:14:07 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 172.876482] nix-grpc-daemon[1578]: time=2026-09-23T04:14:07.479Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 172.878494] nix-grpc-daemon[1578]: time=2026-09-23T04:14:07.480Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)" lb # [ 172.897487] niks3-server[1102]: 2026/09/23 04:14:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign worker1 # [ 172.905399] nix-grpc-daemon[1578]: time=2026-09-23T04:14:07.507Z level=INFO msg="Uploading 1 narinfos" lb # [ 172.912815] niks3-server[1102]: 2026/09/23 04:14:07 INFO Signed narinfos id=57 count=1 lb # [ 172.921532] niks3-server[1102]: 2026/09/23 04:14:07 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst lb # [ 172.936231] niks3-server[1102]: 2026/09/23 04:14:07 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls worker1 # [ 172.944280] nix-grpc-daemon[1578]: time=2026-09-23T04:14:07.546Z level=INFO msg="Upload complete. (236ms)" lb # [ 172.942412] niks3-server[1102]: 2026/09/23 04:14:07 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete lb # [ 172.951994] niks3-server[1102]: 2026/09/23 04:14:07 INFO Completed upload id=57 lb # [ 172.955510] niks3-server[1102]: 2026/09/23 04:14:07 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo worker1 # [ 172.954872] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=640 worker1 # [ 172.969634] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 172.978271] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 172.982363] nix-daemon[849]: reaped child process 1682, status = succeeded client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051... worker1 # [ 172.994459] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) client # farm-b> node-a: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-during-drain worker1 # [ 174.105667] nix-daemon[849]: reaped child process 1691, status = succeeded worker1 # [ 174.178712] nix-daemon[849]: accepted connection from pid 1707, user nix-grpc-daemon (trusted) worker1 # [ 174.198061] nix-daemon[849]: reaped child process 1710, status = succeeded lb # [ 174.201450] niks3-server[1102]: 2026/09/23 04:14:08 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 174.211576] nix-grpc-daemon[1578]: time=2026-09-23T04:14:08.813Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 174.213314] nix-grpc-daemon[1578]: time=2026-09-23T04:14:08.813Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)" lb # [ 174.240099] niks3-server[1102]: 2026/09/23 04:14:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign worker1 # [ 174.248989] nix-grpc-daemon[1578]: time=2026-09-23T04:14:08.851Z level=INFO msg="Uploading 1 narinfos" lb # [ 174.254544] niks3-server[1102]: 2026/09/23 04:14:08 INFO Signed narinfos id=58 count=1 lb # [ 174.266570] niks3-server[1102]: 2026/09/23 04:14:08 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls lb # [ 174.275931] niks3-server[1102]: 2026/09/23 04:14:08 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker1 # [ 174.285975] nix-grpc-daemon[1578]: time=2026-09-23T04:14:08.888Z level=INFO msg="Upload complete. (277ms)" lb # [ 174.287434] niks3-server[1102]: 2026/09/23 04:14:08 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst lb # [ 174.293593] niks3-server[1102]: 2026/09/23 04:14:08 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete lb # [ 174.300547] niks3-server[1102]: 2026/09/23 04:14:08 INFO Completed upload id=58 lb # [ 174.302819] niks3-server[1102]: 2026/09/23 04:14:08 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo worker1 # [ 174.301844] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=1 worker1 # [ 174.320821] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:08Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=0 worker1 # [ 174.325449] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:08Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 174.330317] nix-daemon[849]: reaped child process 1689, status = succeeded worker1 # [ 174.359566] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 174.374632] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 174.388310] nix-grpc-daemon[1578]: time=2026-09-23T04:14:08.990Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... worker1 # [ 174.399203] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'... worker1 # [ 174.426377] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 174.589157] nix-daemon[849]: accepted connection from pid 1716, user nix-grpc-daemon (trusted) worker1 # [ 174.605413] nix-daemon[849]: reaped child process 1719, status = succeeded lb # [ 174.613790] niks3-server[1102]: 2026/09/23 04:14:09 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 174.621539] nix-grpc-daemon[1578]: time=2026-09-23T04:14:09.223Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 174.623091] nix-grpc-daemon[1578]: time=2026-09-23T04:14:09.223Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)" lb # [ 174.640101] niks3-server[1102]: 2026/09/23 04:14:09 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls worker1 # [ 174.650655] nix-grpc-daemon[1578]: time=2026-09-23T04:14:09.253Z level=INFO msg="Uploading 1 narinfos" lb # [ 174.655325] niks3-server[1102]: 2026/09/23 04:14:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign lb # [ 174.666672] niks3-server[1102]: 2026/09/23 04:14:09 INFO Signed narinfos id=59 count=1 lb # [ 174.675795] niks3-server[1102]: 2026/09/23 04:14:09 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst worker1 # [ 174.681961] nix-grpc-daemon[1578]: time=2026-09-23T04:14:09.284Z level=INFO msg="Upload complete. (238ms)" lb # [ 174.684862] niks3-server[1102]: 2026/09/23 04:14:09 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete lb # [ 174.689299] niks3-server[1102]: 2026/09/23 04:14:09 INFO Completed upload id=59 lb # [ 174.692704] niks3-server[1102]: 2026/09/23 04:14:09 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo worker1 # [ 174.692060] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=768 worker1 # [ 174.700617] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 174.712259] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 174.720292] nix-daemon[849]: reaped child process 1714, status = succeeded client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051... worker1 # [ 174.725418] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) client # farm-top> node-a: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-during-drain worker1 # [ 175.604185] nix-daemon[849]: reaped child process 1723, status = succeeded worker1 # [ 175.699583] nix-daemon[849]: accepted connection from pid 1739, user nix-grpc-daemon (trusted) worker1 # [ 175.714507] nix-daemon[849]: reaped child process 1742, status = succeeded lb # [ 175.725785] niks3-server[1102]: 2026/09/23 04:14:10 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 175.734425] nix-grpc-daemon[1578]: time=2026-09-23T04:14:10.335Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 175.736243] nix-grpc-daemon[1578]: time=2026-09-23T04:14:10.335Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)" lb # [ 175.765351] niks3-server[1102]: 2026/09/23 04:14:10 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker1 # [ 175.774165] nix-grpc-daemon[1578]: time=2026-09-23T04:14:10.376Z level=INFO msg="Uploading 1 narinfos" lb # [ 175.778964] niks3-server[1102]: 2026/09/23 04:14:10 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst lb # [ 175.789790] niks3-server[1102]: 2026/09/23 04:14:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign worker1 # [ 175.800155] nix-grpc-daemon[1578]: time=2026-09-23T04:14:10.402Z level=INFO msg="Upload complete. (264ms)" lb # [ 175.802133] niks3-server[1102]: 2026/09/23 04:14:10 INFO Signed narinfos id=60 count=1 lb # [ 175.805922] niks3-server[1102]: 2026/09/23 04:14:10 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls lb # [ 175.814442] niks3-server[1102]: 2026/09/23 04:14:10 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete lb # [ 175.821280] niks3-server[1102]: 2026/09/23 04:14:10 INFO Completed upload id=60 worker1 # [ 175.816216] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=1 lb # [ 175.823999] niks3-server[1102]: 2026/09/23 04:14:10 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo worker1 # [ 175.830672] nix-daemon[849]: reaped child process 1721, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 5.03 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.18 seconds) worker2: must succeed: pkill -f 'read -t [9]0 x' worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds) client: waiting for success: ! systemctl is-active sw worker2 # [ 176.204450] nix-daemon[840]: reaped child process 1788, status = succeeded worker2 # [ 176.319240] nix-daemon[840]: accepted connection from pid 1965, user nix-grpc-daemon (trusted) worker2 # [ 176.336393] nix-daemon[840]: reaped child process 1968, status = succeeded lb # [ 176.339730] niks3-server[1102]: 2026/09/23 04:14:10 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 176.352165] nix-grpc-daemon[1733]: time=2026-09-23T04:14:10.930Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 176.353962] nix-grpc-daemon[1733]: time=2026-09-23T04:14:10.930Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)" lb # [ 176.384673] niks3-server[1102]: 2026/09/23 04:14:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign worker2 # [ 176.396594] nix-grpc-daemon[1733]: time=2026-09-23T04:14:10.975Z level=INFO msg="Uploading 1 narinfos" lb # [ 176.398464] niks3-server[1102]: 2026/09/23 04:14:11 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls lb # [ 176.406201] niks3-server[1102]: 2026/09/23 04:14:11 INFO Signed narinfos id=61 count=1 lb # [ 176.410905] niks3-server[1102]: 2026/09/23 04:14:11 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst lb # [ 176.418885] niks3-server[1102]: 2026/09/23 04:14:11 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv lb # [ 176.422782] niks3-server[1102]: 2026/09/23 04:14:11 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete worker2 # [ 176.426202] nix-grpc-daemon[1733]: time=2026-09-23T04:14:11.005Z level=INFO msg="Upload complete. (242ms)" lb # [ 176.427249] niks3-server[1102]: 2026/09/23 04:14:11 INFO Completed upload id=61 lb # [ 176.431385] niks3-server[1102]: 2026/09/23 04:14:11 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo worker2 # [ 176.441066] nix-grpc-daemon[1651]: ts=2026-09-23T04:14:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56656 duration_s=15 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=1 worker2 # [ 176.455768] nix-daemon[840]: reaped child process 1786, status = succeeded client # [ 176.474737] systemd[1]: sw.service: Deactivated successfully. client # [ 176.480275] systemd[1]: sw.service: Consumed 192ms CPU time over 16.381s wall clock time, 41.2M memory peak, 7.7K incoming IP traffic, 8.7K outgoing IP traffic. worker2 # [ 176.717909] nix-grpc-daemon[1651]: ts=2026-09-23T04:14:11Z level=info event=signal_exit client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds) client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; } client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.05 seconds) worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service worker2 # [ 177.325794] systemd[1]: Starting scheduler-token.service... worker2 # [ 177.398830] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 177.403559] systemd[1]: Finished scheduler-token.service. worker2 # [ 177.405276] systemd[1]: scheduler-token.service: Consumed 41ms CPU time over 76ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker1 # [ 177.725212] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:12Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:50608 worker2 # [ 177.733021] nix-grpc-daemon[1651]: ts=2026-09-23T04:14:12Z level=info event=scheduler_restarting worker1 # [ 177.729917] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:12Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 177.754445] nix-grpc-daemon[1651]: ts=2026-09-23T04:14:12Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 177.781826] nix-daemon[840]: reaped child process 1704, status = succeeded worker2 # [ 178.306959] nix-grpc-daemon[1651]: E0000 00:00:1790136852.885855 1697 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true worker2 # [ 178.313667] nix-grpc-daemon[1651]: E0000 00:00:1790136852.886193 1697 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker2 # [ 179.750443] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 179.752354] systemd[1]: nix-grpc-daemon.service: Consumed 773ms CPU time over 30.314s wall clock time, 35.9M memory peak, 22.7K incoming IP traffic, 36.8K outgoing IP traffic. worker2 # [ 179.778431] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 179.899716] envoy[906]: conn peer=[2001:db8:1::1]:52318 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9608 tx=4427 ms=4782 lb # [ 179.904759] envoy[906]: conn peer=[2001:db8:1::1]:57912 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3064 tx=1578 ms=16168 lb # [ 179.909575] envoy[906]: conn peer=[2001:db8:1::4]:36050 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=7599 tx=679 ms=27969 worker2 # [ 179.926039] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:14Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 179.928475] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:14Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 179.932428] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:14Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 179.938957] nix-grpc-daemon[1985]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 179.948078] nix-grpc-daemon[1985]: E0000 00:00:1790136854.516051 1985 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 179.950440] nix-grpc-daemon[1985]: E0000 00:00:1790136854.518120 1985 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 179.955732] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:14Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 179.980255] nix-grpc-daemon[1985]: E0000 00:00:1790136854.533070 2025 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 179.982656] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 179.983465] nix-grpc-daemon[1985]: E0000 00:00:1790136854.550957 2025 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.75 seconds) lb: waiting until exactly no endpoints of aarch64-linux are unhealthy worker2 # [ 179.996994] nix-grpc-daemon[1985]: E0000 00:00:1790136854.551200 2025 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 179.999433] nix-grpc-daemon[1985]: E0000 00:00:1790136854.574108 2025 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 180.008483] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:14Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 180.008370] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:14Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:50608 worker1 # [ 180.008823] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:14Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.14 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 21.22 seconds) subtest: queue gauges show a burst while every build blocks lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) client: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]' client # [ 181.492738] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto --impure --expr "map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ \"q0\" \"q1\" \"q2\" \"q3\" ]". client # Running as unit: burst.service; invocation ID: 8405c488ba6448a5b1dcbbfcfa4741b0 client: (finished: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]', in 0.09 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker2 # [ 181.695656] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:16Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 181.710809] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) client # [ 181.730296] nix[1551]: don't know how to build these paths: client # [ 181.731751] nix[1551]: /nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv client # [ 181.733965] nix[1551]: /nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv client # [ 181.735476] nix[1551]: /nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv client # [ 181.737973] nix[1551]: /nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker1 # [ 181.745097] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:16Z level=debug event=want client=5 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv cached=0 worker1 # [ 181.745438] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:16Z level=debug event=assigned drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 181.745564] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:16Z level=debug event=want client=5 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv cached=0 worker1 # [ 181.745693] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:16Z level=debug event=assigned drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv worker=192.168.1.4:50051 clients=1 worker1 # [ 181.745832] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:16Z level=debug event=want client=5 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv cached=0 worker1 # [ 181.745961] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:16Z level=debug event=want client=5 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv cached=0 worker2 # [ 181.792125] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 worker1 # [ 181.788990] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 181.804613] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker2 # [ 181.808604] nix-grpc-daemon[1985]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 76 ms (attempt 1/5) client # [ 181.823764] nix[1551]: copying 1 paths to 'lb:50051'... worker1 # [ 181.818815] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 client # [ 181.826444] nix[1551]: copying path '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' to 'lb:50051'... worker1 # [ 181.833593] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker2 # [ 181.893925] nix-grpc-daemon[1985]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 99 ms (attempt 2/5) worker1 # [ 181.987137] nix-daemon[849]: accepted connection from pid 1757, user nix-grpc-daemon (trusted) worker2 # [ 181.994914] nix-grpc-daemon[1985]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 264 ms (attempt 3/5) worker1 # [ 182.005921] nix-daemon[849]: reaped child process 1760, status = succeeded lb # [ 182.010756] niks3-server[1102]: 2026/09/23 04:14:16 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 182.019339] nix-grpc-daemon[1578]: time=2026-09-23T04:14:16.621Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 182.020974] nix-grpc-daemon[1578]: time=2026-09-23T04:14:16.621Z level=INFO msg="Uploading d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv (416B)" lb # [ 182.042228] niks3-server[1102]: 2026/09/23 04:14:16 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.ls worker1 # [ 182.059843] nix-grpc-daemon[1578]: time=2026-09-23T04:14:16.662Z level=INFO msg="Uploading 1 narinfos" lb # [ 182.060621] niks3-server[1102]: 2026/09/23 04:14:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign lb # [ 182.071834] niks3-server[1102]: 2026/09/23 04:14:16 INFO Signed narinfos id=62 count=1 lb # [ 182.087122] niks3-server[1102]: 2026/09/23 04:14:16 INFO Registered completed upload object_key=nar/06ismykzsix0w29371dm1hbzzf3ljf9jn7dyh8skp5wqfkkw0vlk.nar.zst lb # [ 182.101022] niks3-server[1102]: 2026/09/23 04:14:16 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete worker1 # [ 182.104447] nix-grpc-daemon[1578]: time=2026-09-23T04:14:16.706Z level=INFO msg="Upload complete. (255ms)" lb # [ 182.108888] niks3-server[1102]: 2026/09/23 04:14:16 INFO Completed upload id=62 lb # [ 182.115236] niks3-server[1102]: 2026/09/23 04:14:16 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.narinfo worker1 # [ 182.116304] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 182.135012] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 182.141593] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 182.146775] nix-daemon[849]: reaped child process 1755, status = succeeded client # [ 182.162501] nix[1551]: building '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' on lb:50051... worker1 # [ 182.159190] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker2 # [ 182.260796] nix-grpc-daemon[1985]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 591 ms (attempt 4/5) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2 # [ 182.856835] nix-grpc-daemon[1985]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 182.864292] nix-grpc-daemon[1985]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 182.879272] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52070 client # [ 182.896976] nix[1551]: copying 1 paths to 'lb:50051'... worker2 # [ 182.894069] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52070 worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) client # [ 182.897237] nix[1551]: copying path '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' to 'lb:50051'... worker2 # [ 182.909876] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) worker2 # [ 182.937362] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:17Z level=info event=niks3_push_started pid=2046 worker2 # [ 183.105247] nix-grpc-daemon[2046]: time=2026-09-23T04:14:17.683Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 183.215776] nix-daemon[840]: accepted connection from pid 2063, user nix-grpc-daemon (trusted) worker2 # [ 183.234062] nix-daemon[840]: reaped child process 2066, status = succeeded lb # [ 183.242596] niks3-server[1102]: 2026/09/23 04:14:17 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 183.258905] nix-grpc-daemon[2046]: time=2026-09-23T04:14:17.837Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 183.262813] nix-grpc-daemon[2046]: time=2026-09-23T04:14:17.837Z level=INFO msg="Uploading dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv (416B)" lb # [ 183.288277] niks3-server[1102]: 2026/09/23 04:14:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign worker2 # [ 183.300103] nix-grpc-daemon[2046]: time=2026-09-23T04:14:17.878Z level=INFO msg="Uploading 1 narinfos" lb # [ 183.301163] niks3-server[1102]: 2026/09/23 04:14:17 INFO Signed narinfos id=63 count=1 lb # [ 183.303530] niks3-server[1102]: 2026/09/23 04:14:17 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.ls worker2 # [ 183.318011] nix-grpc-daemon[2046]: time=2026-09-23T04:14:17.897Z level=INFO msg="Upload complete. (209ms)" lb # [ 183.316659] niks3-server[1102]: 2026/09/23 04:14:17 INFO Registered completed upload object_key=nar/06mdwsnf16qmfi86y8r33asi4vcrp5z7kdxqki87psgfpgvvpdki.nar.zst worker2 # [ 183.324213] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52070 duration_s=0 paths=1 nar_bytes_in=416 lb # [ 183.323033] niks3-server[1102]: 2026/09/23 04:14:17 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete lb # [ 183.327335] niks3-server[1102]: 2026/09/23 04:14:17 INFO Completed upload id=63 lb # [ 183.329598] niks3-server[1102]: 2026/09/23 04:14:17 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.narinfo worker2 # [ 183.335064] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 183.341058] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) worker2 # [ 183.345445] nix-daemon[840]: reaped child process 2044, status = succeeded client # [ 183.355321] nix[1551]: building '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' on lb:50051... worker2 # [ 183.356887] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) worker1 # [ 184.459401] nix-daemon[849]: reaped child process 1764, status = succeeded worker1 # [ 184.540106] nix-daemon[849]: accepted connection from pid 1792, user nix-grpc-daemon (trusted) worker1 # [ 184.558476] nix-daemon[849]: reaped child process 1795, status = succeeded lb # [ 184.566111] niks3-server[1102]: 2026/09/23 04:14:19 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 184.572437] nix-grpc-daemon[1578]: time=2026-09-23T04:14:19.174Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 184.574110] nix-grpc-daemon[1578]: time=2026-09-23T04:14:19.174Z level=INFO msg="Uploading k6zivhvzgmlyxfmy64fzd759bxlhf9b5-slow-q0 (120B)" lb # [ 184.601970] niks3-server[1102]: 2026/09/23 04:14:19 INFO Registered completed upload object_key=nar/0xr809jj32p5n6mnc6i7lqsz48wrm5f9jp14m77k724041711iq7.nar.zst worker1 # [ 184.619966] nix-grpc-daemon[1578]: time=2026-09-23T04:14:19.222Z level=INFO msg="Uploading 1 narinfos" lb # [ 184.620774] niks3-server[1102]: 2026/09/23 04:14:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign lb # [ 184.629934] niks3-server[1102]: 2026/09/23 04:14:19 INFO Signed narinfos id=64 count=1 lb # [ 184.639398] niks3-server[1102]: 2026/09/23 04:14:19 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete worker1 # [ 184.647294] nix-grpc-daemon[1578]: time=2026-09-23T04:14:19.249Z level=INFO msg="Upload complete. (272ms)" lb # [ 184.649313] niks3-server[1102]: 2026/09/23 04:14:19 INFO Registered completed upload object_key=log/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv lb # [ 184.660945] niks3-server[1102]: 2026/09/23 04:14:19 INFO Completed upload id=64 lb # [ 184.664622] niks3-server[1102]: 2026/09/23 04:14:19 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.narinfo worker1 # [ 184.663375] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=2 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv assign_id=7 outputs=1 lb # [ 184.668798] niks3-server[1102]: 2026/09/23 04:14:19 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.ls worker1 # [ 184.676356] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:19Z level=debug event=assigned drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 184.682059] nix-daemon[849]: reaped child process 1762, status = succeeded worker1 # [ 184.703912] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 184.718640] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 client # [ 184.737288] nix[1551]: copying 1 paths to 'lb:50051'... worker1 # [ 184.733161] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 client # [ 184.742367] nix[1551]: copying path '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' to 'lb:50051'... worker1 # [ 184.746180] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 184.898540] nix-daemon[849]: accepted connection from pid 1799, user nix-grpc-daemon (trusted) worker1 # [ 184.913870] nix-daemon[849]: reaped child process 1802, status = succeeded lb # [ 184.921132] niks3-server[1102]: 2026/09/23 04:14:19 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 184.929012] nix-grpc-daemon[1578]: time=2026-09-23T04:14:19.531Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 184.930557] nix-grpc-daemon[1578]: time=2026-09-23T04:14:19.531Z level=INFO msg="Uploading jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv (416B)" worker1 # [ 184.957122] nix-grpc-daemon[1578]: time=2026-09-23T04:14:19.559Z level=INFO msg="Uploading 1 narinfos" lb # [ 184.950952] niks3-server[1102]: 2026/09/23 04:14:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign lb # [ 184.966390] niks3-server[1102]: 2026/09/23 04:14:19 INFO Signed narinfos id=65 count=1 lb # [ 184.974507] niks3-server[1102]: 2026/09/23 04:14:19 INFO Registered completed upload object_key=nar/1fyfjzlx8vymsfmdv5cv8ymgjkr18fwm737qvys29plwwiwmyvwr.nar.zst lb # [ 184.991041] niks3-server[1102]: 2026/09/23 04:14:19 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.ls worker1 # [ 184.998510] nix-grpc-daemon[1578]: time=2026-09-23T04:14:19.601Z level=INFO msg="Upload complete. (237ms)" lb # [ 185.000683] niks3-server[1102]: 2026/09/23 04:14:19 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete lb # [ 185.006457] niks3-server[1102]: 2026/09/23 04:14:19 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.narinfo lb # [ 185.012336] niks3-server[1102]: 2026/09/23 04:14:19 INFO Completed upload id=65 worker1 # [ 185.008296] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 185.024634] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 185.034420] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 185.038302] nix-daemon[849]: reaped child process 1797, status = succeeded worker1 # [ 185.049511] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) client # [ 185.053153] nix[1551]: building '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' on lb:50051... worker2 # [ 185.586728] nix-daemon[840]: reaped child process 2070, status = succeeded worker2 # [ 185.686435] nix-daemon[840]: accepted connection from pid 2095, user nix-grpc-daemon (trusted) worker2 # [ 185.707305] nix-daemon[840]: reaped child process 2098, status = succeeded lb # [ 185.704748] niks3-server[1102]: 2026/09/23 04:14:20 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 185.719145] nix-grpc-daemon[2046]: time=2026-09-23T04:14:20.296Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 185.721619] nix-grpc-daemon[2046]: time=2026-09-23T04:14:20.296Z level=INFO msg="Uploading y3qkgrgq2zzpgvi2rf5chq94i627wkdb-slow-q3 (120B)" worker2 # [ 185.758164] nix-grpc-daemon[2046]: time=2026-09-23T04:14:20.336Z level=INFO msg="Uploading 1 narinfos" lb # [ 185.747071] niks3-server[1102]: 2026/09/23 04:14:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign lb # [ 185.762061] niks3-server[1102]: 2026/09/23 04:14:20 INFO Signed narinfos id=66 count=1 lb # [ 185.771172] niks3-server[1102]: 2026/09/23 04:14:20 INFO Registered completed upload object_key=log/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv lb # [ 185.780708] niks3-server[1102]: 2026/09/23 04:14:20 INFO Registered completed upload object_key=nar/1v1grzkvw3yj09x0dkdm2avihvqvnri156qzp768hk85zbw7lkn9.nar.zst lb # [ 185.790391] niks3-server[1102]: 2026/09/23 04:14:20 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.ls worker2 # [ 185.798173] nix-grpc-daemon[2046]: time=2026-09-23T04:14:20.377Z level=INFO msg="Upload complete. (230ms)" lb # [ 185.796419] niks3-server[1102]: 2026/09/23 04:14:20 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete lb # [ 185.800287] niks3-server[1102]: 2026/09/23 04:14:20 INFO Completed upload id=66 lb # [ 185.804768] niks3-server[1102]: 2026/09/23 04:14:20 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.narinfo worker2 # [ 185.806974] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 duration_s=2 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv assign_id=8 outputs=1 worker1 # [ 185.804889] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:20Z level=debug event=assigned drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 185.825955] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 185.830503] nix-daemon[840]: reaped child process 2068, status = succeeded worker2 # [ 185.837476] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52070 client # [ 185.849302] nix[1551]: copying 1 paths to 'lb:50051'... worker2 # [ 185.848651] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52070 client # [ 185.853344] nix[1551]: copying path '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' to 'lb:50051'... worker2 # [ 185.859494] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) worker2 # [ 185.987273] nix-daemon[840]: accepted connection from pid 2102, user nix-grpc-daemon (trusted) worker2 # [ 186.005527] nix-daemon[840]: reaped child process 2105, status = succeeded lb # [ 186.007680] niks3-server[1102]: 2026/09/23 04:14:20 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 186.018649] nix-grpc-daemon[2046]: time=2026-09-23T04:14:20.597Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 186.020203] nix-grpc-daemon[2046]: time=2026-09-23T04:14:20.597Z level=INFO msg="Uploading pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv (416B)" lb # [ 186.039381] niks3-server[1102]: 2026/09/23 04:14:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign worker2 # [ 186.051280] nix-grpc-daemon[2046]: time=2026-09-23T04:14:20.630Z level=INFO msg="Uploading 1 narinfos" lb # [ 186.050347] niks3-server[1102]: 2026/09/23 04:14:20 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.ls lb # [ 186.059337] niks3-server[1102]: 2026/09/23 04:14:20 INFO Signed narinfos id=67 count=1 worker2 # [ 186.070239] nix-grpc-daemon[2046]: time=2026-09-23T04:14:20.649Z level=INFO msg="Upload complete. (199ms)" lb # [ 186.068348] niks3-server[1102]: 2026/09/23 04:14:20 INFO Registered completed upload object_key=nar/1k21jrz411xc9gkca2pnr9mrjq1s3bg3zhsam3nnv4j066dgzgm7.nar.zst lb # [ 186.077168] niks3-server[1102]: 2026/09/23 04:14:20 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete worker2 # [ 186.078369] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52070 duration_s=0 paths=1 nar_bytes_in=416 lb # [ 186.082560] niks3-server[1102]: 2026/09/23 04:14:20 INFO Completed upload id=67 lb # [ 186.084945] niks3-server[1102]: 2026/09/23 04:14:20 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.narinfo worker2 # [ 186.091553] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 186.100327] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) worker2 # [ 186.103687] nix-daemon[840]: reaped child process 2100, status = succeeded client # [ 186.115068] nix[1551]: building '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' on lb:50051... worker2 # [ 186.116154] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) worker1 # [ 186.688465] nix-daemon[849]: reaped child process 1806, status = succeeded worker1 # [ 186.782372] nix-daemon[849]: accepted connection from pid 1831, user nix-grpc-daemon (trusted) worker1 # [ 186.797398] nix-daemon[849]: reaped child process 1834, status = succeeded lb # [ 186.806823] niks3-server[1102]: 2026/09/23 04:14:21 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 186.816369] nix-grpc-daemon[1578]: time=2026-09-23T04:14:21.417Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 186.817828] nix-grpc-daemon[1578]: time=2026-09-23T04:14:21.417Z level=INFO msg="Uploading gjyi8j90pk5flrsqqdhj6h132pcb7n33-slow-q2 (120B)" lb # [ 186.848935] niks3-server[1102]: 2026/09/23 04:14:21 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.ls lb # [ 186.860877] niks3-server[1102]: 2026/09/23 04:14:21 INFO Registered completed upload object_key=nar/15imdx7zyy6mzjjv6qvdzywh0d9w1n6ikc0rq4rq95aw9x6swfzj.nar.zst worker1 # [ 186.870247] nix-grpc-daemon[1578]: time=2026-09-23T04:14:21.472Z level=INFO msg="Uploading 1 narinfos" lb # [ 186.874020] niks3-server[1102]: 2026/09/23 04:14:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign lb # [ 186.882590] niks3-server[1102]: 2026/09/23 04:14:21 INFO Signed narinfos id=68 count=1 lb # [ 186.887163] niks3-server[1102]: 2026/09/23 04:14:21 INFO Registered completed upload object_key=log/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv worker1 # [ 186.897012] nix-grpc-daemon[1578]: time=2026-09-23T04:14:21.499Z level=INFO msg="Upload complete. (234ms)" lb # [ 186.899010] niks3-server[1102]: 2026/09/23 04:14:21 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete lb # [ 186.905363] niks3-server[1102]: 2026/09/23 04:14:21 INFO Completed upload id=68 lb # [ 186.907775] niks3-server[1102]: 2026/09/23 04:14:21 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.narinfo worker1 # [ 186.909087] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=1 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv assign_id=9 outputs=1 worker1 # [ 186.924167] nix-daemon[849]: reaped child process 1804, status = succeeded worker2 # [ 187.950620] nix-daemon[840]: reaped child process 2109, status = succeeded worker2 # [ 188.051276] nix-daemon[840]: accepted connection from pid 2134, user nix-grpc-daemon (trusted) worker2 # [ 188.070389] nix-daemon[840]: reaped child process 2137, status = succeeded lb # [ 188.073000] niks3-server[1102]: 2026/09/23 04:14:22 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 188.086380] nix-grpc-daemon[2046]: time=2026-09-23T04:14:22.664Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 188.087842] nix-grpc-daemon[2046]: time=2026-09-23T04:14:22.664Z level=INFO msg="Uploading l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx-slow-q1 (120B)" lb # [ 188.113515] niks3-server[1102]: 2026/09/23 04:14:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign worker2 # [ 188.124539] nix-grpc-daemon[2046]: time=2026-09-23T04:14:22.703Z level=INFO msg="Uploading 1 narinfos" lb # [ 188.128457] niks3-server[1102]: 2026/09/23 04:14:22 INFO Signed narinfos id=69 count=1 lb # [ 188.131212] niks3-server[1102]: 2026/09/23 04:14:22 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.ls lb # [ 188.145808] niks3-server[1102]: 2026/09/23 04:14:22 INFO Registered completed upload object_key=log/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker2 # [ 188.152806] nix-grpc-daemon[2046]: time=2026-09-23T04:14:22.732Z level=INFO msg="Upload complete. (225ms)" lb # [ 188.152413] niks3-server[1102]: 2026/09/23 04:14:22 INFO Registered completed upload object_key=nar/0c1c367r3msa4fz6imwp70zxi4fxvrkb86v3hv921wsrf13yd8v3.nar.zst lb # [ 188.157394] niks3-server[1102]: 2026/09/23 04:14:22 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete lb # [ 188.160836] niks3-server[1102]: 2026/09/23 04:14:22 INFO Completed upload id=69 lb # [ 188.163562] niks3-server[1102]: 2026/09/23 04:14:22 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.narinfo worker2 # [ 188.164737] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 duration_s=2 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv assign_id=10 outputs=1 worker2 # [ 188.176074] nix-daemon[840]: reaped child process 2107, status = succeeded client # [ 188.206949] systemd[1]: burst.service: Deactivated successfully. client # [ 188.211952] systemd[1]: burst.service: Consumed 230ms CPU time over 6.715s wall clock time, 51.7M memory peak, 14K incoming IP traffic, 18.9K outgoing IP traffic. client: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; } client: (finished: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) (finished: subtest: queue gauges show a burst while every build blocks, in 8.14 seconds) subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1 worker2 # [ 189.617013] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:52070 worker1 # [ 189.617916] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 189.661062] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:24Z level=debug event=want client=6 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=0 worker1 # [ 189.668124] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:24Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 189.674047] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:24Z level=debug event=want client=6 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=0 worker1 # [ 189.679219] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:24Z level=debug event=want client=6 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=0 worker1 # [ 189.712585] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 189.727860] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 189.752422] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 189.767319] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) lb # [ 189.901448] envoy[906]: conn peer=[2001:db8:1::1]:36102 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9714 tx=5223 ms=6492 worker1 # [ 189.924919] nix-daemon[849]: accepted connection from pid 1851, user nix-grpc-daemon (trusted) worker1 # [ 189.942723] nix-daemon[849]: reaped child process 1854, status = succeeded lb # [ 189.951387] niks3-server[1102]: 2026/09/23 04:14:24 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 189.961977] nix-grpc-daemon[1578]: time=2026-09-23T04:14:24.563Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 189.963534] nix-grpc-daemon[1578]: time=2026-09-23T04:14:24.563Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)" lb # [ 189.992613] niks3-server[1102]: 2026/09/23 04:14:24 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls worker1 # [ 190.002158] nix-grpc-daemon[1578]: time=2026-09-23T04:14:24.604Z level=INFO msg="Uploading 1 narinfos" lb # [ 190.004322] niks3-server[1102]: 2026/09/23 04:14:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign lb # [ 190.013403] niks3-server[1102]: 2026/09/23 04:14:24 INFO Signed narinfos id=70 count=1 lb # [ 190.023975] niks3-server[1102]: 2026/09/23 04:14:24 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst lb # [ 190.039545] niks3-server[1102]: 2026/09/23 04:14:24 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete worker1 # [ 190.043598] nix-grpc-daemon[1578]: time=2026-09-23T04:14:24.646Z level=INFO msg="Upload complete. (259ms)" lb # [ 190.046689] niks3-server[1102]: 2026/09/23 04:14:24 INFO Completed upload id=70 lb # [ 190.050290] niks3-server[1102]: 2026/09/23 04:14:24 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo worker1 # [ 190.056619] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 190.072179] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 190.079155] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 190.085235] nix-daemon[849]: reaped child process 1849, status = succeeded worker1 # [ 190.101128] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 191.533525] nix-daemon[849]: reaped child process 1858, status = succeeded worker1 # [ 191.635942] nix-daemon[849]: accepted connection from pid 1874, user nix-grpc-daemon (trusted) worker1 # [ 191.651711] nix-daemon[849]: reaped child process 1877, status = succeeded lb # [ 191.662274] niks3-server[1102]: 2026/09/23 04:14:26 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 191.668381] nix-grpc-daemon[1578]: time=2026-09-23T04:14:26.270Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 191.669835] nix-grpc-daemon[1578]: time=2026-09-23T04:14:26.270Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)" lb # [ 191.705583] niks3-server[1102]: 2026/09/23 04:14:26 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker1 # [ 191.714928] nix-grpc-daemon[1578]: time=2026-09-23T04:14:26.317Z level=INFO msg="Uploading 1 narinfos" lb # [ 191.720355] niks3-server[1102]: 2026/09/23 04:14:26 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst lb # [ 191.728573] niks3-server[1102]: 2026/09/23 04:14:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign lb # [ 191.731727] niks3-server[1102]: 2026/09/23 04:14:26 INFO Signed narinfos id=71 count=1 lb # [ 191.736987] niks3-server[1102]: 2026/09/23 04:14:26 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls worker1 # [ 191.743310] nix-grpc-daemon[1578]: time=2026-09-23T04:14:26.345Z level=INFO msg="Upload complete. (230ms)" lb # [ 191.744979] niks3-server[1102]: 2026/09/23 04:14:26 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete lb # [ 191.748861] niks3-server[1102]: 2026/09/23 04:14:26 INFO Completed upload id=71 lb # [ 191.752278] niks3-server[1102]: 2026/09/23 04:14:26 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo worker1 # [ 191.759378] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=11 outputs=1 worker1 # [ 191.773639] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:26Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 191.783887] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 191.789950] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:26Z level=debug event=want client=6 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=0 worker1 # [ 191.795558] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 191.812266] nix-daemon[849]: reaped child process 1856, status = succeeded worker1 # [ 191.814034] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 191.818638] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 191.946257] nix-daemon[849]: accepted connection from pid 1883, user nix-grpc-daemon (trusted) worker1 # [ 191.961569] nix-daemon[849]: reaped child process 1886, status = succeeded lb # [ 191.969486] niks3-server[1102]: 2026/09/23 04:14:26 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 191.976689] nix-grpc-daemon[1578]: time=2026-09-23T04:14:26.578Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 191.978174] nix-grpc-daemon[1578]: time=2026-09-23T04:14:26.578Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)" lb # [ 191.998523] niks3-server[1102]: 2026/09/23 04:14:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign worker1 # [ 192.004703] nix-grpc-daemon[1578]: time=2026-09-23T04:14:26.607Z level=INFO msg="Uploading 1 narinfos" lb # [ 192.012931] niks3-server[1102]: 2026/09/23 04:14:26 INFO Signed narinfos id=72 count=1 lb # [ 192.020656] niks3-server[1102]: 2026/09/23 04:14:26 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst lb # [ 192.036110] niks3-server[1102]: 2026/09/23 04:14:26 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls worker1 # [ 192.039650] nix-grpc-daemon[1578]: time=2026-09-23T04:14:26.642Z level=INFO msg="Upload complete. (217ms)" lb # [ 192.039973] niks3-server[1102]: 2026/09/23 04:14:26 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete lb # [ 192.049331] niks3-server[1102]: 2026/09/23 04:14:26 INFO Completed upload id=72 worker1 # [ 192.049020] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 192.052793] niks3-server[1102]: 2026/09/23 04:14:26 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo worker1 # [ 192.062025] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 192.067645] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 192.072293] nix-daemon[849]: reaped child process 1881, status = succeeded worker1 # [ 192.083988] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 193.111476] nix-daemon[849]: reaped child process 1890, status = succeeded worker1 # [ 193.203233] nix-daemon[849]: accepted connection from pid 1906, user nix-grpc-daemon (trusted) worker1 # [ 193.220311] nix-daemon[849]: reaped child process 1909, status = succeeded lb # [ 193.229713] niks3-server[1102]: 2026/09/23 04:14:27 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 193.236531] nix-grpc-daemon[1578]: time=2026-09-23T04:14:27.838Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 193.238005] nix-grpc-daemon[1578]: time=2026-09-23T04:14:27.838Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)" lb # [ 193.271030] niks3-server[1102]: 2026/09/23 04:14:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign worker1 # [ 193.278834] nix-grpc-daemon[1578]: time=2026-09-23T04:14:27.881Z level=INFO msg="Uploading 1 narinfos" lb # [ 193.285009] niks3-server[1102]: 2026/09/23 04:14:27 INFO Signed narinfos id=73 count=1 lb # [ 193.287976] niks3-server[1102]: 2026/09/23 04:14:27 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst worker1 # [ 193.304310] nix-grpc-daemon[1578]: time=2026-09-23T04:14:27.906Z level=INFO msg="Upload complete. (211ms)" lb # [ 193.308504] niks3-server[1102]: 2026/09/23 04:14:27 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker1 # [ 193.312730] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=12 outputs=1 lb # [ 193.315396] niks3-server[1102]: 2026/09/23 04:14:27 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete lb # [ 193.324954] niks3-server[1102]: 2026/09/23 04:14:27 INFO Completed upload id=73 lb # [ 193.328330] niks3-server[1102]: 2026/09/23 04:14:27 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls worker1 # [ 193.328457] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:27Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=1 lb # [ 193.335013] niks3-server[1102]: 2026/09/23 04:14:27 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo worker1 # [ 193.335189] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:27Z level=debug event=want client=6 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=0 worker1 # [ 193.338840] nix-daemon[849]: reaped child process 1888, status = succeeded worker1 # [ 193.366940] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 193.381668] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 193.395497] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 193.409396] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 193.555189] nix-daemon[849]: accepted connection from pid 1915, user nix-grpc-daemon (trusted) worker1 # [ 193.573067] nix-daemon[849]: reaped child process 1918, status = succeeded lb # [ 193.577341] niks3-server[1102]: 2026/09/23 04:14:28 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 193.584741] nix-grpc-daemon[1578]: time=2026-09-23T04:14:28.187Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 193.586510] nix-grpc-daemon[1578]: time=2026-09-23T04:14:28.187Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)" worker1 # [ 193.606156] nix-grpc-daemon[1578]: time=2026-09-23T04:14:28.208Z level=INFO msg="Uploading 1 narinfos" lb # [ 193.607402] niks3-server[1102]: 2026/09/23 04:14:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign lb # [ 193.620164] niks3-server[1102]: 2026/09/23 04:14:28 INFO Signed narinfos id=74 count=1 lb # [ 193.623345] niks3-server[1102]: 2026/09/23 04:14:28 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls worker1 # [ 193.637053] nix-grpc-daemon[1578]: time=2026-09-23T04:14:28.239Z level=INFO msg="Upload complete. (216ms)" lb # [ 193.639213] niks3-server[1102]: 2026/09/23 04:14:28 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst worker1 # [ 193.644146] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 193.648125] niks3-server[1102]: 2026/09/23 04:14:28 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete lb # [ 193.651667] niks3-server[1102]: 2026/09/23 04:14:28 INFO Completed upload id=74 lb # [ 193.654384] niks3-server[1102]: 2026/09/23 04:14:28 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo worker1 # [ 193.657008] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 193.664684] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 193.667307] nix-daemon[849]: reaped child process 1913, status = succeeded worker1 # [ 193.677006] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 194.781248] nix-daemon[849]: reaped child process 1922, status = succeeded worker1 # [ 194.844634] nix-daemon[849]: accepted connection from pid 1938, user nix-grpc-daemon (trusted) worker1 # [ 194.862164] nix-daemon[849]: reaped child process 1941, status = succeeded lb # [ 194.871409] niks3-server[1102]: 2026/09/23 04:14:29 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 194.880478] nix-grpc-daemon[1578]: time=2026-09-23T04:14:29.481Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 194.882530] nix-grpc-daemon[1578]: time=2026-09-23T04:14:29.481Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)" lb # [ 194.909940] niks3-server[1102]: 2026/09/23 04:14:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign worker1 # [ 194.916864] nix-grpc-daemon[1578]: time=2026-09-23T04:14:29.519Z level=INFO msg="Uploading 1 narinfos" lb # [ 194.924574] niks3-server[1102]: 2026/09/23 04:14:29 INFO Signed narinfos id=75 count=1 lb # [ 194.927836] niks3-server[1102]: 2026/09/23 04:14:29 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst lb # [ 194.950330] niks3-server[1102]: 2026/09/23 04:14:29 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker1 # [ 194.955195] nix-grpc-daemon[1578]: time=2026-09-23T04:14:29.557Z level=INFO msg="Upload complete. (266ms)" lb # [ 194.958510] niks3-server[1102]: 2026/09/23 04:14:29 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls lb # [ 194.962152] niks3-server[1102]: 2026/09/23 04:14:29 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete lb # [ 194.966614] niks3-server[1102]: 2026/09/23 04:14:29 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo lb # [ 194.970797] niks3-server[1102]: 2026/09/23 04:14:29 INFO Completed upload id=75 worker1 # [ 194.966938] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=13 outputs=1 worker1 # [ 194.987297] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:29Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 194.992955] nix-daemon[849]: reaped child process 1920, status = succeeded worker1 # [ 195.014454] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 195.022563] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:29Z level=debug event=want client=6 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=0 worker1 # [ 195.034794] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 195.052922] nix-grpc-daemon[1578]: time=2026-09-23T04:14:29.655Z level=INFO msg="All 1 paths already cached" worker1 # [ 195.064898] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 195.077451] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 195.225365] nix-daemon[849]: accepted connection from pid 1947, user nix-grpc-daemon (trusted) worker1 # [ 195.241250] nix-daemon[849]: reaped child process 1950, status = succeeded lb # [ 195.248881] niks3-server[1102]: 2026/09/23 04:14:29 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 195.257362] nix-grpc-daemon[1578]: time=2026-09-23T04:14:29.859Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 195.258934] nix-grpc-daemon[1578]: time=2026-09-23T04:14:29.859Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)" lb # [ 195.280622] niks3-server[1102]: 2026/09/23 04:14:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign worker1 # [ 195.285939] nix-grpc-daemon[1578]: time=2026-09-23T04:14:29.888Z level=INFO msg="Uploading 1 narinfos" lb # [ 195.294989] niks3-server[1102]: 2026/09/23 04:14:29 INFO Signed narinfos id=76 count=1 lb # [ 195.302315] niks3-server[1102]: 2026/09/23 04:14:29 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls lb # [ 195.312050] niks3-server[1102]: 2026/09/23 04:14:29 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst lb # [ 195.316651] niks3-server[1102]: 2026/09/23 04:14:29 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete worker1 # [ 195.319910] nix-grpc-daemon[1578]: time=2026-09-23T04:14:29.922Z level=INFO msg="Upload complete. (228ms)" lb # [ 195.322820] niks3-server[1102]: 2026/09/23 04:14:29 INFO Completed upload id=76 lb # [ 195.326037] niks3-server[1102]: 2026/09/23 04:14:29 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo worker1 # [ 195.328325] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 195.343671] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 195.354602] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 195.357325] nix-daemon[849]: reaped child process 1945, status = succeeded worker1 # [ 195.370827] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 196.440849] nix-daemon[849]: reaped child process 1954, status = succeeded worker1 # [ 196.544291] nix-daemon[849]: accepted connection from pid 1970, user nix-grpc-daemon (trusted) worker1 # [ 196.564173] nix-daemon[849]: reaped child process 1973, status = succeeded lb # [ 196.571452] niks3-server[1102]: 2026/09/23 04:14:31 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 196.586141] nix-grpc-daemon[1578]: time=2026-09-23T04:14:31.186Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 196.588269] nix-grpc-daemon[1578]: time=2026-09-23T04:14:31.186Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)" lb # [ 196.619394] niks3-server[1102]: 2026/09/23 04:14:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign worker1 # [ 196.623750] nix-grpc-daemon[1578]: time=2026-09-23T04:14:31.226Z level=INFO msg="Uploading 1 narinfos" lb # [ 196.630044] niks3-server[1102]: 2026/09/23 04:14:31 INFO Signed narinfos id=77 count=1 lb # [ 196.637003] niks3-server[1102]: 2026/09/23 04:14:31 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst lb # [ 196.644822] niks3-server[1102]: 2026/09/23 04:14:31 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls lb # [ 196.651223] niks3-server[1102]: 2026/09/23 04:14:31 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv lb # [ 196.655727] niks3-server[1102]: 2026/09/23 04:14:31 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete worker1 # [ 196.653504] nix-grpc-daemon[1578]: time=2026-09-23T04:14:31.255Z level=INFO msg="Upload complete. (272ms)" lb # [ 196.658710] niks3-server[1102]: 2026/09/23 04:14:31 INFO Completed upload id=77 lb # [ 196.663558] niks3-server[1102]: 2026/09/23 04:14:31 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo worker1 # [ 196.670974] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=14 outputs=1 worker1 # [ 196.688265] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:31Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 196.694691] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 196.699810] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:31Z level=debug event=want client=6 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=0 worker1 # [ 196.704594] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 196.709135] nix-daemon[849]: reaped child process 1952, status = succeeded worker1 # [ 196.710874] nix-grpc-daemon[1578]: time=2026-09-23T04:14:31.310Z level=INFO msg="All 1 paths already cached" worker1 # [ 196.714275] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 196.728471] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 196.851214] nix-daemon[849]: accepted connection from pid 1979, user nix-grpc-daemon (trusted) worker1 # [ 196.868275] nix-daemon[849]: reaped child process 1982, status = succeeded lb # [ 196.875762] niks3-server[1102]: 2026/09/23 04:14:31 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 196.884061] nix-grpc-daemon[1578]: time=2026-09-23T04:14:31.486Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 196.885547] nix-grpc-daemon[1578]: time=2026-09-23T04:14:31.486Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)" lb # [ 196.908136] niks3-server[1102]: 2026/09/23 04:14:31 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls worker1 # [ 196.921918] nix-grpc-daemon[1578]: time=2026-09-23T04:14:31.524Z level=INFO msg="Uploading 1 narinfos" lb # [ 196.922747] niks3-server[1102]: 2026/09/23 04:14:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign lb # [ 196.934833] niks3-server[1102]: 2026/09/23 04:14:31 INFO Signed narinfos id=78 count=1 lb # [ 196.942044] niks3-server[1102]: 2026/09/23 04:14:31 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst lb # [ 196.954818] niks3-server[1102]: 2026/09/23 04:14:31 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete lb # [ 196.960559] niks3-server[1102]: 2026/09/23 04:14:31 INFO Completed upload id=78 worker1 # [ 196.958764] nix-grpc-daemon[1578]: time=2026-09-23T04:14:31.561Z level=INFO msg="Upload complete. (224ms)" lb # [ 196.963562] niks3-server[1102]: 2026/09/23 04:14:31 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo worker1 # [ 196.968330] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 196.986284] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 196.993487] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 196.998873] nix-daemon[849]: reaped child process 1977, status = succeeded worker1 # [ 197.010211] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 197.636994] nix-daemon[849]: reaped child process 1986, status = succeeded worker1 # [ 197.738496] nix-daemon[849]: accepted connection from pid 2002, user nix-grpc-daemon (trusted) worker1 # [ 197.758618] nix-daemon[849]: reaped child process 2005, status = succeeded lb # [ 197.759142] niks3-server[1102]: 2026/09/23 04:14:32 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 197.770317] nix-grpc-daemon[1578]: time=2026-09-23T04:14:32.371Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 197.771801] nix-grpc-daemon[1578]: time=2026-09-23T04:14:32.371Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)" lb # [ 197.800097] niks3-server[1102]: 2026/09/23 04:14:32 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst lb # [ 197.812378] niks3-server[1102]: 2026/09/23 04:14:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign worker1 # [ 197.815562] nix-grpc-daemon[1578]: time=2026-09-23T04:14:32.417Z level=INFO msg="Uploading 1 narinfos" lb # [ 197.821055] niks3-server[1102]: 2026/09/23 04:14:32 INFO Signed narinfos id=79 count=1 lb # [ 197.826139] niks3-server[1102]: 2026/09/23 04:14:32 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls lb # [ 197.833189] niks3-server[1102]: 2026/09/23 04:14:32 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv lb # [ 197.841589] niks3-server[1102]: 2026/09/23 04:14:32 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete worker1 # [ 197.845046] nix-grpc-daemon[1578]: time=2026-09-23T04:14:32.447Z level=INFO msg="Upload complete. (220ms)" lb # [ 197.847236] niks3-server[1102]: 2026/09/23 04:14:32 INFO Completed upload id=79 lb # [ 197.851010] niks3-server[1102]: 2026/09/23 04:14:32 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo worker1 # [ 197.860191] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=15 outputs=1 worker1 # [ 197.876791] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:32Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 197.882774] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 197.888559] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:32Z level=debug event=want client=6 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=0 worker1 # [ 197.892831] nix-daemon[849]: reaped child process 1984, status = succeeded worker1 # [ 197.911237] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 197.924831] nix-grpc-daemon[1578]: time=2026-09-23T04:14:32.527Z level=INFO msg="All 1 paths already cached" worker1 # [ 197.933533] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 197.947815] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 198.087801] nix-daemon[849]: accepted connection from pid 2011, user nix-grpc-daemon (trusted) worker1 # [ 198.107590] nix-daemon[849]: reaped child process 2014, status = succeeded lb # [ 198.112149] niks3-server[1102]: 2026/09/23 04:14:32 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 198.120155] nix-grpc-daemon[1578]: time=2026-09-23T04:14:32.722Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 198.122272] nix-grpc-daemon[1578]: time=2026-09-23T04:14:32.724Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)" lb # [ 198.142937] niks3-server[1102]: 2026/09/23 04:14:32 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst worker1 # [ 198.154174] nix-grpc-daemon[1578]: time=2026-09-23T04:14:32.756Z level=INFO msg="Uploading 1 narinfos" lb # [ 198.157245] niks3-server[1102]: 2026/09/23 04:14:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign lb # [ 198.167211] niks3-server[1102]: 2026/09/23 04:14:32 INFO Signed narinfos id=80 count=1 worker1 # [ 198.176103] nix-grpc-daemon[1578]: time=2026-09-23T04:14:32.778Z level=INFO msg="Upload complete. (216ms)" lb # [ 198.177136] niks3-server[1102]: 2026/09/23 04:14:32 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls lb # [ 198.181483] niks3-server[1102]: 2026/09/23 04:14:32 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete lb # [ 198.188559] niks3-server[1102]: 2026/09/23 04:14:32 INFO Completed upload id=80 worker1 # [ 198.184135] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 198.190446] niks3-server[1102]: 2026/09/23 04:14:32 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo worker1 # [ 198.199395] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 198.206125] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 198.210755] nix-daemon[849]: reaped child process 2009, status = succeeded worker1 # [ 198.221477] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 198.871192] nix-daemon[849]: reaped child process 2018, status = succeeded worker1 # [ 198.968563] nix-daemon[849]: accepted connection from pid 2034, user nix-grpc-daemon (trusted) worker1 # [ 198.985700] nix-daemon[849]: reaped child process 2037, status = succeeded lb # [ 198.997818] niks3-server[1102]: 2026/09/23 04:14:33 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 199.006192] nix-grpc-daemon[1578]: time=2026-09-23T04:14:33.606Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 199.007758] nix-grpc-daemon[1578]: time=2026-09-23T04:14:33.606Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)" lb # [ 199.033937] niks3-server[1102]: 2026/09/23 04:14:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign worker1 # [ 199.039785] nix-grpc-daemon[1578]: time=2026-09-23T04:14:33.642Z level=INFO msg="Uploading 1 narinfos" lb # [ 199.050268] niks3-server[1102]: 2026/09/23 04:14:33 INFO Signed narinfos id=81 count=1 lb # [ 199.056842] niks3-server[1102]: 2026/09/23 04:14:33 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst worker1 # [ 199.065303] nix-grpc-daemon[1578]: time=2026-09-23T04:14:33.667Z level=INFO msg="Upload complete. (232ms)" lb # [ 199.068240] niks3-server[1102]: 2026/09/23 04:14:33 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls lb # [ 199.073140] niks3-server[1102]: 2026/09/23 04:14:33 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv lb # [ 199.076789] niks3-server[1102]: 2026/09/23 04:14:33 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete lb # [ 199.082546] niks3-server[1102]: 2026/09/23 04:14:33 INFO Completed upload id=81 worker1 # [ 199.078609] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=16 outputs=1 lb # [ 199.086084] niks3-server[1102]: 2026/09/23 04:14:33 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo worker1 # [ 199.092585] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:33Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 199.099557] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 199.105292] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:33Z level=debug event=want client=6 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=0 worker1 # [ 199.111085] nix-daemon[849]: reaped child process 2016, status = succeeded worker1 # [ 199.114478] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 199.119550] nix-grpc-daemon[1578]: time=2026-09-23T04:14:33.720Z level=INFO msg="All 2 paths already cached" worker1 # [ 199.123121] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 199.136215] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 199.256094] nix-daemon[849]: accepted connection from pid 2043, user nix-grpc-daemon (trusted) worker1 # [ 199.273068] nix-daemon[849]: reaped child process 2046, status = succeeded lb # [ 199.279385] niks3-server[1102]: 2026/09/23 04:14:33 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 199.286905] nix-grpc-daemon[1578]: time=2026-09-23T04:14:33.889Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 199.288513] nix-grpc-daemon[1578]: time=2026-09-23T04:14:33.889Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)" lb # [ 199.308555] niks3-server[1102]: 2026/09/23 04:14:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign worker1 # [ 199.313273] nix-grpc-daemon[1578]: time=2026-09-23T04:14:33.915Z level=INFO msg="Uploading 1 narinfos" lb # [ 199.321775] niks3-server[1102]: 2026/09/23 04:14:33 INFO Signed narinfos id=82 count=1 lb # [ 199.330466] niks3-server[1102]: 2026/09/23 04:14:33 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls lb # [ 199.337644] niks3-server[1102]: 2026/09/23 04:14:33 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst worker1 # [ 199.344496] nix-grpc-daemon[1578]: time=2026-09-23T04:14:33.947Z level=INFO msg="Upload complete. (202ms)" lb # [ 199.346387] niks3-server[1102]: 2026/09/23 04:14:33 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete worker1 # [ 199.352137] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 199.355597] niks3-server[1102]: 2026/09/23 04:14:33 INFO Completed upload id=82 lb # [ 199.358134] niks3-server[1102]: 2026/09/23 04:14:33 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo worker1 # [ 199.363722] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 199.370075] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 199.374661] nix-daemon[849]: reaped child process 2041, status = succeeded worker1 # [ 199.385568] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 200.651463] nix-daemon[849]: reaped child process 2050, status = succeeded worker1 # [ 200.772833] nix-daemon[849]: accepted connection from pid 2066, user nix-grpc-daemon (trusted) worker1 # [ 200.789997] nix-daemon[849]: reaped child process 2069, status = succeeded lb # [ 200.798824] niks3-server[1102]: 2026/09/23 04:14:35 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 200.811375] nix-grpc-daemon[1578]: time=2026-09-23T04:14:35.412Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 200.813458] nix-grpc-daemon[1578]: time=2026-09-23T04:14:35.412Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)" lb # [ 200.838954] niks3-server[1102]: 2026/09/23 04:14:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign worker1 # [ 200.846694] nix-grpc-daemon[1578]: time=2026-09-23T04:14:35.448Z level=INFO msg="Uploading 1 narinfos" lb # [ 200.855317] niks3-server[1102]: 2026/09/23 04:14:35 INFO Signed narinfos id=83 count=1 lb # [ 200.865067] niks3-server[1102]: 2026/09/23 04:14:35 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls lb # [ 200.874416] niks3-server[1102]: 2026/09/23 04:14:35 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst worker1 # [ 200.887574] nix-grpc-daemon[1578]: time=2026-09-23T04:14:35.490Z level=INFO msg="Upload complete. (290ms)" lb # [ 200.889010] niks3-server[1102]: 2026/09/23 04:14:35 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv lb # [ 200.894681] niks3-server[1102]: 2026/09/23 04:14:35 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete lb # [ 200.899561] niks3-server[1102]: 2026/09/23 04:14:35 INFO Completed upload id=83 lb # [ 200.903447] niks3-server[1102]: 2026/09/23 04:14:35 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo worker1 # [ 200.901952] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=17 outputs=1 worker1 # [ 200.920534] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:35Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 200.926453] nix-daemon[849]: reaped child process 2048, status = succeeded worker1 # [ 200.947522] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 200.963671] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 200.978330] nix-grpc-daemon[1578]: time=2026-09-23T04:14:35.580Z level=INFO msg="All 2 paths already cached" worker1 # [ 200.988111] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 201.001261] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 201.148283] nix-daemon[849]: accepted connection from pid 2073, user nix-grpc-daemon (trusted) worker1 # [ 201.165803] nix-daemon[849]: reaped child process 2076, status = succeeded lb # [ 201.173722] niks3-server[1102]: 2026/09/23 04:14:35 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 201.181404] nix-grpc-daemon[1578]: time=2026-09-23T04:14:35.783Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 201.182930] nix-grpc-daemon[1578]: time=2026-09-23T04:14:35.783Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)" lb # [ 201.200671] niks3-server[1102]: 2026/09/23 04:14:35 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst worker1 # [ 201.210624] nix-grpc-daemon[1578]: time=2026-09-23T04:14:35.813Z level=INFO msg="Uploading 1 narinfos" lb # [ 201.213861] niks3-server[1102]: 2026/09/23 04:14:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign lb # [ 201.227033] niks3-server[1102]: 2026/09/23 04:14:35 INFO Signed narinfos id=84 count=1 lb # [ 201.232286] niks3-server[1102]: 2026/09/23 04:14:35 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls worker1 # [ 201.242634] nix-grpc-daemon[1578]: time=2026-09-23T04:14:35.845Z level=INFO msg="Upload complete. (226ms)" lb # [ 201.244344] niks3-server[1102]: 2026/09/23 04:14:35 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete lb # [ 201.248434] niks3-server[1102]: 2026/09/23 04:14:35 INFO Completed upload id=84 lb # [ 201.251898] niks3-server[1102]: 2026/09/23 04:14:35 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo worker1 # [ 201.252283] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 201.267552] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 201.276510] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 201.279520] nix-daemon[849]: reaped child process 2071, status = succeeded worker1 # [ 201.290643] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 202.144353] nix-daemon[849]: reaped child process 2080, status = succeeded worker1 # [ 202.270497] nix-daemon[849]: accepted connection from pid 2096, user nix-grpc-daemon (trusted) worker1 # [ 202.285590] nix-daemon[849]: reaped child process 2099, status = succeeded lb # [ 202.295255] niks3-server[1102]: 2026/09/23 04:14:36 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 202.308718] nix-grpc-daemon[1578]: time=2026-09-23T04:14:36.907Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 202.310569] nix-grpc-daemon[1578]: time=2026-09-23T04:14:36.907Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)" lb # [ 202.342072] niks3-server[1102]: 2026/09/23 04:14:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign worker1 # [ 202.348857] nix-grpc-daemon[1578]: time=2026-09-23T04:14:36.951Z level=INFO msg="Uploading 1 narinfos" lb # [ 202.356400] niks3-server[1102]: 2026/09/23 04:14:36 INFO Signed narinfos id=85 count=1 lb # [ 202.363730] niks3-server[1102]: 2026/09/23 04:14:36 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv lb # [ 202.376264] niks3-server[1102]: 2026/09/23 04:14:36 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls lb # [ 202.379807] niks3-server[1102]: 2026/09/23 04:14:36 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst lb # [ 202.390698] niks3-server[1102]: 2026/09/23 04:14:36 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete lb # [ 202.397818] niks3-server[1102]: 2026/09/23 04:14:37 INFO Completed upload id=85 worker1 # [ 202.386810] nix-grpc-daemon[1578]: time=2026-09-23T04:14:36.989Z level=INFO msg="Upload complete. (284ms)" lb # [ 202.401479] niks3-server[1102]: 2026/09/23 04:14:37 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo worker1 # [ 202.402066] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=18 outputs=1 worker1 # [ 202.420293] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:36Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 202.426990] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 202.432665] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 202.437632] nix-daemon[849]: reaped child process 2078, status = succeeded worker1 # [ 202.440642] nix-grpc-daemon[1578]: time=2026-09-23T04:14:37.042Z level=INFO msg="All 2 paths already cached" worker1 # [ 202.449220] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 202.462152] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 202.594138] nix-daemon[849]: accepted connection from pid 2103, user nix-grpc-daemon (trusted) worker1 # [ 202.612659] nix-daemon[849]: reaped child process 2106, status = succeeded lb # [ 202.618120] niks3-server[1102]: 2026/09/23 04:14:37 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 202.625566] nix-grpc-daemon[1578]: time=2026-09-23T04:14:37.227Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 202.627075] nix-grpc-daemon[1578]: time=2026-09-23T04:14:37.227Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)" lb # [ 202.650555] niks3-server[1102]: 2026/09/23 04:14:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/86/sign worker1 # [ 202.655311] nix-grpc-daemon[1578]: time=2026-09-23T04:14:37.257Z level=INFO msg="Uploading 1 narinfos" lb # [ 202.663846] niks3-server[1102]: 2026/09/23 04:14:37 INFO Signed narinfos id=86 count=1 lb # [ 202.670025] niks3-server[1102]: 2026/09/23 04:14:37 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls worker1 # [ 202.679643] nix-grpc-daemon[1578]: time=2026-09-23T04:14:37.282Z level=INFO msg="Upload complete. (207ms)" lb # [ 202.681176] niks3-server[1102]: 2026/09/23 04:14:37 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst lb # [ 202.689258] niks3-server[1102]: 2026/09/23 04:14:37 INFO Received complete upload request method=POST path=/api/pending_closures/86/complete worker1 # [ 202.688185] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 202.692729] niks3-server[1102]: 2026/09/23 04:14:37 INFO Completed upload id=86 lb # [ 202.694862] niks3-server[1102]: 2026/09/23 04:14:37 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo worker1 # [ 202.697929] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 worker1 # [ 202.705615] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 202.708447] nix-daemon[849]: reaped child process 2101, status = succeeded worker1 # [ 202.717780] nix-daemon[849]: accepted connection from pid 1477, user nix-grpc-daemon (trusted) worker1 # [ 203.611507] nix-daemon[849]: reaped child process 2110, status = succeeded worker1 # [ 203.696511] nix-daemon[849]: accepted connection from pid 2126, user nix-grpc-daemon (trusted) worker1 # [ 203.712897] nix-daemon[849]: reaped child process 2129, status = succeeded lb # [ 203.720861] niks3-server[1102]: 2026/09/23 04:14:38 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 203.730848] nix-grpc-daemon[1578]: time=2026-09-23T04:14:38.332Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 203.732624] nix-grpc-daemon[1578]: time=2026-09-23T04:14:38.332Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)" lb # [ 203.761913] niks3-server[1102]: 2026/09/23 04:14:38 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls worker1 # [ 203.771164] nix-grpc-daemon[1578]: time=2026-09-23T04:14:38.373Z level=INFO msg="Uploading 1 narinfos" lb # [ 203.774962] niks3-server[1102]: 2026/09/23 04:14:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/87/sign lb # [ 203.785422] niks3-server[1102]: 2026/09/23 04:14:38 INFO Signed narinfos id=87 count=1 lb # [ 203.791206] niks3-server[1102]: 2026/09/23 04:14:38 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv lb # [ 203.798185] niks3-server[1102]: 2026/09/23 04:14:38 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst lb # [ 203.808285] niks3-server[1102]: 2026/09/23 04:14:38 INFO Received complete upload request method=POST path=/api/pending_closures/87/complete lb # [ 203.813366] niks3-server[1102]: 2026/09/23 04:14:38 INFO Completed upload id=87 worker1 # [ 203.804579] nix-grpc-daemon[1578]: time=2026-09-23T04:14:38.407Z level=INFO msg="Upload complete. (276ms)" lb # [ 203.817197] niks3-server[1102]: 2026/09/23 04:14:38 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo worker1 # [ 203.817730] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50618 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=19 outputs=1 worker1 # [ 203.837031] nix-daemon[849]: reaped child process 2108, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 14.49 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]' client # [ 204.013996] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]". client # Running as unit: gpu.service; invocation ID: 22268d7f132d4580b40413ca5015c79d client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics client # [ 204.219059] nix[1623]: don't know how to build these paths: worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) client # [ 204.219997] nix[1623]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv worker2 # [ 204.217211] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:38Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 204.225245] nix-grpc-daemon[1985]: ts=2026-09-23T04:14:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52070 client # [ 204.250686] nix[1623]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip}, waiting up to 30s for a worker worker1 # [ 204.245008] nix-grpc-daemon[1477]: ts=2026-09-23T04:14:38Z level=debug event=want client=7 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) client: waiting for success: systemctl show -p Result gpu | grep -q exit-code lb # [ 209.902119] envoy[906]: conn peer=[2001:db8:1::1]:36116 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27080 tx=12299 ms=14224 client # [ 234.256637] nix[1623]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip} client # [ 234.269609] nix[1623]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed client # [ 234.311691] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE client # [ 234.319486] systemd[1]: gpu.service: Failed with result 'exit-code'. client # [ 234.323997] systemd[1]: gpu.service: Consumed 200ms CPU time over 30.303s wall clock time, 37.4M memory peak, 5.6K incoming IP traffic, 5.2K outgoing IP traffic. client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.03 seconds) client: must succeed: journalctl -u gpu -o cat client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 45.24 seconds) subtest: low disk drains a worker and builds go to the other worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.16 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free' worker1 # [ 236.607904] nix-grpc-daemon[1477]: ts=2026-09-23T04:15:11Z level=info event=unhealthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.11 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2 client # don't know how to build these paths: client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker2 # [ 237.136113] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 237.144711] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52070 worker1 # [ 237.168479] nix-grpc-daemon[1477]: ts=2026-09-23T04:15:11Z level=debug event=want client=8 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=0 worker1 # [ 237.174365] nix-grpc-daemon[1477]: ts=2026-09-23T04:15:11Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 237.219355] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 237.236398] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52070 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'... worker2 # [ 237.254899] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 237.271667] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) worker2 # [ 237.439740] nix-daemon[840]: accepted connection from pid 2145, user nix-grpc-daemon (trusted) worker2 # [ 237.460394] nix-daemon[840]: reaped child process 2148, status = succeeded lb # [ 237.462359] niks3-server[1102]: 2026/09/23 04:15:12 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 237.480313] nix-grpc-daemon[2046]: time=2026-09-23T04:15:12.058Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 237.482391] nix-grpc-daemon[2046]: time=2026-09-23T04:15:12.058Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)" lb # [ 237.512582] niks3-server[1102]: 2026/09/23 04:15:12 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst worker2 # [ 237.529391] nix-grpc-daemon[2046]: time=2026-09-23T04:15:12.108Z level=INFO msg="Uploading 1 narinfos" lb # [ 237.526184] niks3-server[1102]: 2026/09/23 04:15:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/88/sign lb # [ 237.536395] niks3-server[1102]: 2026/09/23 04:15:12 INFO Signed narinfos id=88 count=1 lb # [ 237.554565] niks3-server[1102]: 2026/09/23 04:15:12 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls lb # [ 237.568671] niks3-server[1102]: 2026/09/23 04:15:12 INFO Received complete upload request method=POST path=/api/pending_closures/88/complete lb # [ 237.576366] niks3-server[1102]: 2026/09/23 04:15:12 INFO Completed upload id=88 worker2 # [ 237.577747] nix-grpc-daemon[2046]: time=2026-09-23T04:15:12.156Z level=INFO msg="Upload complete. (286ms)" lb # [ 237.580506] niks3-server[1102]: 2026/09/23 04:15:12 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo worker2 # [ 237.588383] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52070 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 237.609115] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 237.622665] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) worker2 # [ 237.627308] nix-daemon[840]: reaped child process 2143, status = succeeded client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051... worker2 # [ 237.643506] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) client # farm-a> worker2-next: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-drain worker2 # [ 238.543422] nix-daemon[840]: reaped child process 2152, status = succeeded worker2 # [ 238.612897] nix-daemon[840]: accepted connection from pid 2168, user nix-grpc-daemon (trusted) worker2 # [ 238.632786] nix-daemon[840]: reaped child process 2171, status = succeeded lb # [ 238.634440] niks3-server[1102]: 2026/09/23 04:15:13 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 238.651728] nix-grpc-daemon[2046]: time=2026-09-23T04:15:13.228Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 238.653678] nix-grpc-daemon[2046]: time=2026-09-23T04:15:13.229Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)" lb # [ 238.688094] niks3-server[1102]: 2026/09/23 04:15:13 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst lb # [ 238.697223] niks3-server[1102]: 2026/09/23 04:15:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/89/sign worker2 # [ 238.708205] nix-grpc-daemon[2046]: time=2026-09-23T04:15:13.286Z level=INFO msg="Uploading 1 narinfos" lb # [ 238.708571] niks3-server[1102]: 2026/09/23 04:15:13 INFO Signed narinfos id=89 count=1 lb # [ 238.715437] niks3-server[1102]: 2026/09/23 04:15:13 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv lb # [ 238.723603] niks3-server[1102]: 2026/09/23 04:15:13 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls lb # [ 238.730736] niks3-server[1102]: 2026/09/23 04:15:13 INFO Received complete upload request method=POST path=/api/pending_closures/89/complete lb # [ 238.737764] niks3-server[1102]: 2026/09/23 04:15:13 INFO Completed upload id=89 worker2 # [ 238.739588] nix-grpc-daemon[2046]: time=2026-09-23T04:15:13.318Z level=INFO msg="Upload complete. (280ms)" lb # [ 238.739953] niks3-server[1102]: 2026/09/23 04:15:13 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo worker2 # [ 238.753325] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=20 outputs=1 worker1 # [ 238.754303] nix-grpc-daemon[1477]: ts=2026-09-23T04:15:13Z level=debug event=want client=8 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=0 worker1 # [ 238.760668] nix-grpc-daemon[1477]: ts=2026-09-23T04:15:13Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 238.776246] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 238.783096] nix-daemon[840]: reaped child process 2150, status = succeeded worker2 # [ 238.791432] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 238.805681] nix-grpc-daemon[2046]: time=2026-09-23T04:15:13.384Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'... worker2 # [ 238.816921] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 238.832572] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) worker2 # [ 238.981602] nix-daemon[840]: accepted connection from pid 2175, user nix-grpc-daemon (trusted) worker2 # [ 239.004586] nix-daemon[840]: reaped child process 2178, status = succeeded lb # [ 239.000639] niks3-server[1102]: 2026/09/23 04:15:13 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 239.015678] nix-grpc-daemon[2046]: time=2026-09-23T04:15:13.594Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 239.017429] nix-grpc-daemon[2046]: time=2026-09-23T04:15:13.594Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)" lb # [ 239.036720] niks3-server[1102]: 2026/09/23 04:15:13 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst worker2 # [ 239.049339] nix-grpc-daemon[2046]: time=2026-09-23T04:15:13.628Z level=INFO msg="Uploading 1 narinfos" lb # [ 239.053302] niks3-server[1102]: 2026/09/23 04:15:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/90/sign lb # [ 239.060729] niks3-server[1102]: 2026/09/23 04:15:13 INFO Signed narinfos id=90 count=1 worker2 # [ 239.073203] nix-grpc-daemon[2046]: time=2026-09-23T04:15:13.652Z level=INFO msg="Upload complete. (226ms)" lb # [ 239.071516] niks3-server[1102]: 2026/09/23 04:15:13 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls lb # [ 239.078199] niks3-server[1102]: 2026/09/23 04:15:13 INFO Received complete upload request method=POST path=/api/pending_closures/90/complete worker2 # [ 239.084133] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52070 duration_s=0 paths=1 nar_bytes_in=632 lb # [ 239.085048] niks3-server[1102]: 2026/09/23 04:15:13 INFO Completed upload id=90 lb # [ 239.087033] niks3-server[1102]: 2026/09/23 04:15:13 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo worker2 # [ 239.095007] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 239.104737] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) worker2 # [ 239.107664] nix-daemon[840]: reaped child process 2173, status = succeeded client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051... worker2 # [ 239.121276] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) client # farm-b> worker2-next: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv lb # [ 239.910213] envoy[906]: conn peer=[2001:db8:1::1]:57642 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=931 tx=613 ms=30071 client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-drain worker2 # [ 240.385827] nix-daemon[840]: reaped child process 2182, status = succeeded worker2 # [ 240.517824] nix-daemon[840]: accepted connection from pid 2198, user nix-grpc-daemon (trusted) worker2 # [ 240.538317] nix-daemon[840]: reaped child process 2201, status = succeeded lb # [ 240.535553] niks3-server[1102]: 2026/09/23 04:15:15 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 240.551793] nix-grpc-daemon[2046]: time=2026-09-23T04:15:15.128Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 240.553636] nix-grpc-daemon[2046]: time=2026-09-23T04:15:15.128Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)" lb # [ 240.584623] niks3-server[1102]: 2026/09/23 04:15:15 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst worker2 # [ 240.599460] nix-grpc-daemon[2046]: time=2026-09-23T04:15:15.178Z level=INFO msg="Uploading 1 narinfos" lb # [ 240.600075] niks3-server[1102]: 2026/09/23 04:15:15 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls lb # [ 240.603492] niks3-server[1102]: 2026/09/23 04:15:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/91/sign lb # [ 240.616750] niks3-server[1102]: 2026/09/23 04:15:15 INFO Signed narinfos id=91 count=1 lb # [ 240.618703] niks3-server[1102]: 2026/09/23 04:15:15 INFO Received complete upload request method=POST path=/api/pending_closures/91/complete worker2 # [ 240.619514] nix-grpc-daemon[2046]: time=2026-09-23T04:15:15.198Z level=INFO msg="Upload complete. (287ms)" lb # [ 240.627489] niks3-server[1102]: 2026/09/23 04:15:15 INFO Completed upload id=91 lb # [ 240.629390] niks3-server[1102]: 2026/09/23 04:15:15 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo worker2 # [ 240.631844] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=21 outputs=1 lb # [ 240.635045] niks3-server[1102]: 2026/09/23 04:15:15 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker1 # [ 240.636618] nix-grpc-daemon[1477]: ts=2026-09-23T04:15:15Z level=debug event=want client=8 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=0 worker1 # [ 240.642988] nix-grpc-daemon[1477]: ts=2026-09-23T04:15:15Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 240.650953] nix-daemon[840]: reaped child process 2180, status = succeeded worker2 # [ 240.687084] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 240.702727] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 240.718401] nix-grpc-daemon[2046]: time=2026-09-23T04:15:15.297Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'... worker2 # [ 240.729987] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 240.744578] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) worker2 # [ 240.909720] nix-daemon[840]: accepted connection from pid 2205, user nix-grpc-daemon (trusted) worker2 # [ 240.933068] nix-daemon[840]: reaped child process 2208, status = succeeded lb # [ 240.934526] niks3-server[1102]: 2026/09/23 04:15:15 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 240.949414] nix-grpc-daemon[2046]: time=2026-09-23T04:15:15.528Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 240.951396] nix-grpc-daemon[2046]: time=2026-09-23T04:15:15.528Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)" worker2 # [ 240.984294] nix-grpc-daemon[2046]: time=2026-09-23T04:15:15.563Z level=INFO msg="Uploading 1 narinfos" lb # [ 240.972304] niks3-server[1102]: 2026/09/23 04:15:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/92/sign lb # [ 240.989385] niks3-server[1102]: 2026/09/23 04:15:15 INFO Signed narinfos id=92 count=1 lb # [ 240.996521] niks3-server[1102]: 2026/09/23 04:15:15 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls lb # [ 241.005901] niks3-server[1102]: 2026/09/23 04:15:15 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst worker2 # [ 241.023472] nix-grpc-daemon[2046]: time=2026-09-23T04:15:15.602Z level=INFO msg="Upload complete. (261ms)" lb # [ 241.021542] niks3-server[1102]: 2026/09/23 04:15:15 INFO Received complete upload request method=POST path=/api/pending_closures/92/complete lb # [ 241.026574] niks3-server[1102]: 2026/09/23 04:15:15 INFO Completed upload id=92 lb # [ 241.030072] niks3-server[1102]: 2026/09/23 04:15:15 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo worker2 # [ 241.036426] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52070 duration_s=0 paths=1 nar_bytes_in=760 worker2 # [ 241.056737] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 worker2 # [ 241.065109] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) worker2 # [ 241.071718] nix-daemon[840]: reaped child process 2203, status = succeeded client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051... client # farm-top> worker2-next: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker2 # [ 241.083031] nix-daemon[840]: accepted connection from pid 1985, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-drain worker2 # [ 242.188693] nix-daemon[840]: reaped child process 2212, status = succeeded worker2 # [ 242.267629] nix-daemon[840]: accepted connection from pid 2228, user nix-grpc-daemon (trusted) worker2 # [ 242.290616] nix-daemon[840]: reaped child process 2231, status = succeeded lb # [ 242.287767] niks3-server[1102]: 2026/09/23 04:15:16 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 242.306813] nix-grpc-daemon[2046]: time=2026-09-23T04:15:16.884Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 242.308700] nix-grpc-daemon[2046]: time=2026-09-23T04:15:16.884Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)" worker2 # [ 242.344421] nix-grpc-daemon[2046]: time=2026-09-23T04:15:16.923Z level=INFO msg="Uploading 1 narinfos" lb # [ 242.331973] niks3-server[1102]: 2026/09/23 04:15:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/93/sign lb # [ 242.347258] niks3-server[1102]: 2026/09/23 04:15:16 INFO Signed narinfos id=93 count=1 lb # [ 242.355751] niks3-server[1102]: 2026/09/23 04:15:16 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls lb # [ 242.364089] niks3-server[1102]: 2026/09/23 04:15:16 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv lb # [ 242.367388] niks3-server[1102]: 2026/09/23 04:15:16 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst worker2 # [ 242.381267] nix-grpc-daemon[2046]: time=2026-09-23T04:15:16.960Z level=INFO msg="Upload complete. (284ms)" lb # [ 242.379477] niks3-server[1102]: 2026/09/23 04:15:16 INFO Received complete upload request method=POST path=/api/pending_closures/93/complete lb # [ 242.383076] niks3-server[1102]: 2026/09/23 04:15:16 INFO Completed upload id=93 lb # [ 242.386573] niks3-server[1102]: 2026/09/23 04:15:16 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo worker2 # [ 242.393185] nix-grpc-daemon[1985]: ts=2026-09-23T04:15:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52070 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=22 outputs=1 worker2 # [ 242.414427] nix-daemon[840]: reaped child process 2210, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 5.53 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.19 seconds) worker1: must succeed: rm /nix/.rw-store/fill worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.16 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free' worker1 # [ 243.612703] nix-grpc-daemon[1477]: ts=2026-09-23T04:15:18Z level=info event=healthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.11 seconds) (finished: subtest: low disk drains a worker and builds go to the other, in 9.35 seconds) (finished: run the VM test script, in 244.90 seconds) test script finished in 245.26s cleanup kill QemuMachine (pid 49) client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) client # [2026-09-23T04:15:19Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-23T04:15:19Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-23T04:15:19Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 50) lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) lb # [2026-09-23T04:15:19Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-23T04:15:19Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-23T04:15:19Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 51) worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker1 # [2026-09-23T04:15:19Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-23T04:15:19Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-23T04:15:19Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 52) worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker2 # [2026-09-23T04:15:20Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-23T04:15:20Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-23T04:15:20Z INFO virtiofsd] Client disconnected, shutting down (finished: cleanup, in 1.08 seconds) additionally exposed symbols: client, lb, worker1, worker2, vlan1, start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh