vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #519
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12client # Disk image does not exist, creating the virtualisation disk image...13client: QEMU running (pid 50)14client # Formatting '/build/vm-state-client/tmp.3JdPeQgcgr', fmt=raw size=107374182415client # mke2fs 1.47.4 (6-Mar-2025)16client # Discarding device blocks: 0/262144 done17client # Creating filesystem with 262144 4k blocks and 65536 inodes18client # Filesystem UUID: 4e9015c0-a04f-40a3-8b31-8366bb04c02419client # Superblock backups stored on blocks:20client # 32768, 98304, 163840, 22937621client # 22client # Allocating group tables: 0/8 done23client # Writing inode tables: 0/8 done24client # Creating journal (8192 blocks): done25client # Writing superblocks and filesystem accounting information: 0/8 done26client # 27client # Virtualisation disk image created.28client # Starting virtiofs daemons...29client # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)30client # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether31client # [2026-09-23T05:11:49Z INFO virtiofsd] Waiting for vhost-user socket connection...32client # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)33client # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether34client # [2026-09-23T05:11:49Z INFO virtiofsd] Waiting for vhost-user socket connection...35client # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)36client # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether37client # [2026-09-23T05:11:49Z INFO virtiofsd] Waiting for vhost-user socket connection...38client # [2026-09-23T05:11:49Z INFO virtiofsd] Client connected, servicing requests39client # [2026-09-23T05:11:49Z INFO virtiofsd] Client connected, servicing requests40client # [2026-09-23T05:11:49Z INFO virtiofsd] Client connected, servicing requests41worker2 # Disk image does not exist, creating the virtualisation disk image...42worker2: QEMU running (pid 52)43worker1 # Disk image does not exist, creating the virtualisation disk image...44worker1: QEMU running (pid 51)45worker1 # Formatting '/build/vm-state-worker1/tmp.wSc80EWke0', fmt=raw size=107374182446worker2 # Formatting '/build/vm-state-worker2/tmp.Y1fP7FGQtb', fmt=raw size=107374182447worker1 # mke2fs 1.47.4 (6-Mar-2025)48worker2 # mke2fs 1.47.4 (6-Mar-2025)49worker1 # Discarding device blocks: 0/262144 done50worker2 # Discarding device blocks: 0/262144 done51worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes52worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes53worker1 # Filesystem UUID: 63669f72-065a-4aa7-9929-31c39caf032a54worker2 # Filesystem UUID: 4aaa835e-3c9e-42bd-bc9c-5f8ae23dc72c55worker1 # Superblock backups stored on blocks:56worker2 # Superblock backups stored on blocks:57worker1 # 32768, 98304, 163840, 22937658worker2 # 32768, 98304, 163840, 22937659worker1 # 60worker2 # 61worker1 # Allocating group tables: 0/8 done62worker2 # Allocating group tables: 0/8 done63worker1 # Writing inode tables: 0/8 done64lb # Disk image does not exist, creating the virtualisation disk image...65lb: QEMU running (pid 49)66worker2 # Writing inode tables: 0/8 done67worker1 # Creating journal (8192 blocks): done68worker2 # Creating journal (8192 blocks): done69worker1 # Writing superblocks and filesystem accounting information: 0/8 done70lb # Formatting '/build/vm-state-lb/tmp.vPgtWf9w9r', fmt=raw size=107374182471(finished: start all VMs, in 0.24 seconds)72worker2 # Writing superblocks and filesystem accounting information: 0/8 done73worker1 # 74lb # mke2fs 1.47.4 (6-Mar-2025)75client: waiting for unit network-addresses-eth1.service76worker2 # 77worker1 # Virtualisation disk image created.78lb # Discarding device blocks: 0/262144 done79client: waiting for the VM to finish booting80worker2 # Virtualisation disk image created.81worker1 # Starting virtiofs daemons...82worker2 # Starting virtiofs daemons...83worker1 # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)84worker2 # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)85worker1 # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether86worker2 # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether87worker1 # [2026-09-23T05:11:49Z INFO virtiofsd] Waiting for vhost-user socket connection...88worker2 # [2026-09-23T05:11:49Z INFO virtiofsd] Waiting for vhost-user socket connection...89worker1 # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)90worker2 # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)91worker1 # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether92worker2 # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether93worker1 # [2026-09-23T05:11:49Z INFO virtiofsd] Waiting for vhost-user socket connection...94worker2 # [2026-09-23T05:11:49Z INFO virtiofsd] Waiting for vhost-user socket connection...95worker1 # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)96worker2 # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)97worker1 # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether98worker2 # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether99worker1 # [2026-09-23T05:11:49Z INFO virtiofsd] Waiting for vhost-user socket connection...100worker2 # [2026-09-23T05:11:49Z INFO virtiofsd] Waiting for vhost-user socket connection...101worker1 # [2026-09-23T05:11:49Z INFO virtiofsd] Client connected, servicing requests102worker2 # [2026-09-23T05:11:49Z INFO virtiofsd] Client connected, servicing requests103worker1 # [2026-09-23T05:11:49Z INFO virtiofsd] Client connected, servicing requests104worker2 # [2026-09-23T05:11:49Z INFO virtiofsd] Client connected, servicing requests105worker1 # [2026-09-23T05:11:49Z INFO virtiofsd] Client connected, servicing requests106worker2 # [2026-09-23T05:11:49Z INFO virtiofsd] Client connected, servicing requests107lb # Creating filesystem with 262144 4k blocks and 65536 inodes108lb # Filesystem UUID: 11894d65-9fd4-44f5-b19c-c059c008b0fb109lb # Superblock backups stored on blocks:110lb # 32768, 98304, 163840, 229376111lb # 112lb # Allocating group tables: 0/8 done113lb # Writing inode tables: 0/8 done114lb # Creating journal (8192 blocks): done115lb # Writing superblocks and filesystem accounting information: 0/8 done116lb # 117lb # Virtualisation disk image created.118lb # Starting virtiofs daemons...119lb # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)120lb # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether121lb # [2026-09-23T05:11:49Z INFO virtiofsd] Waiting for vhost-user socket connection...122lb # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)123lb # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether124lb # [2026-09-23T05:11:49Z INFO virtiofsd] Waiting for vhost-user socket connection...125lb # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)126lb # [2026-09-23T05:11:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether127lb # [2026-09-23T05:11:49Z INFO virtiofsd] Waiting for vhost-user socket connection...128lb # [2026-09-23T05:11:49Z INFO virtiofsd] Client connected, servicing requests129lb # [2026-09-23T05:11:49Z INFO virtiofsd] Client connected, servicing requests130lb # [2026-09-23T05:11:49Z INFO virtiofsd] Client connected, servicing requests131client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)132worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)133lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)134worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)135client # 136client # 137client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00138worker1 # 139worker1 # 140client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...141worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00142client # 143client # 144client # 145client # 146lb # 147lb # 148client # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00149lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00150worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...151worker1 # 152worker1 # 153worker1 # 154worker1 # 155worker1 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00156lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...157lb # 158lb # 159lb # 160lb # 161client # Press Ctrl-B to configure iPXE (PCI 00:05.0)...162client # 163client # 164lb # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00165worker1 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...166worker1 # 167worker1 # 168worker2 # 169worker2 # 170lb # Press Ctrl-B to configure iPXE (PCI 00:05.0)...171lb # 172lb # 173worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00174worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...175worker2 # 176worker2 # 177worker2 # 178worker2 # 179worker2 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00180worker2 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...181worker2 # 182worker2 # 183client # Booting from ROM...184worker1 # Booting from ROM...185lb # Booting from ROM...186worker2 # Booting from ROM...187client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026188lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026189client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/qgir8rpawilvsh39a93frcz2w73apwjr-nixos-system-client-test/init regInfo=/nix/store/lkz8fx5m8vmi31dy1j6w4dxp7rzq8kxy-closure-info/registration console=ttyS0,115200n8 console=tty0190client # [ 0.000000] BIOS-provided physical RAM map:191client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable192client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved193lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/ggsnp37phfgq4b11zirffk4vxgicg7h8-nixos-system-lb-test/init regInfo=/nix/store/94h72crvqzbw5sjsl7v7v31gd90khbjb-closure-info/registration console=ttyS0,115200n8 console=tty0194client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved195worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026196lb # [ 0.000000] BIOS-provided physical RAM map:197client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable198lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable199client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved200lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved201client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved202lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved203client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved204lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable205worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/1xcjwc0c3jblpxjbzzik759i2a779qhn-nixos-system-worker1-test/init regInfo=/nix/store/lplh0yfdzmjpfyfgf19r87dfbm8ls4wa-closure-info/registration console=ttyS0,115200n8 console=tty0206client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved207lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved208worker1 # [ 0.000000] BIOS-provided physical RAM map:209client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved210lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved211worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable212client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved213lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved214worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved215client # [ 0.000000] NX (Execute Disable) protection: active216client # [ 0.000000] APIC: Static calls initialized217lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved218worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved219client # [ 0.000000] SMBIOS 2.8 present.220lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved221worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable222client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014223worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved224lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved225client # [ 0.000000] DMI: Memory slots populated: 1/1226lb # [ 0.000000] NX (Execute Disable) protection: active227worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved228client # [ 0.000000] Hypervisor detected: KVM229lb # [ 0.000000] APIC: Static calls initialized230lb # [ 0.000000] SMBIOS 2.8 present.231client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000232worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved233client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00234worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved235lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014236client # [ 0.000000] kvm-clock: using sched offset of 448075220 cycles237lb # [ 0.000000] DMI: Memory slots populated: 1/1238worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved239lb # [ 0.000000] Hypervisor detected: KVM240worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved241lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000242client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns243worker1 # [ 0.000000] NX (Execute Disable) protection: active244lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00245client # [ 0.000006] tsc: Detected 2400.012 MHz processor246worker1 # [ 0.000000] APIC: Static calls initialized247lb # [ 0.000000] kvm-clock: using sched offset of 468300240 cycles248worker1 # [ 0.000000] SMBIOS 2.8 present.249client # [ 0.000813] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000250lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns251worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014252worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026253client # [ 0.000842] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs254worker1 # [ 0.000000] DMI: Memory slots populated: 1/1255lb # [ 0.000006] tsc: Detected 2400.012 MHz processor256worker1 # [ 0.000000] Hypervisor detected: KVM257lb # [ 0.000813] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000258client # [ 0.000845] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT259worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000260lb # [ 0.000840] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs261client # [ 0.002732] found SMP MP-table at [mem 0x000f5450-0x000f545f]262worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00263client # [ 0.002746] Using GB pages for direct mapping264lb # [ 0.000843] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT265worker1 # [ 0.000001] kvm-clock: using sched offset of 451601251 cycles266lb # [ 0.002741] found SMP MP-table at [mem 0x000f5450-0x000f545f]267client # [ 0.002861] RAMDISK: [mem 0x5e370000-0x5ffcffff]268lb # [ 0.002754] Using GB pages for direct mapping269worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/vjdbc1032z3354xwbcs1ba0jyv7imbfs-nixos-system-worker2-test/init regInfo=/nix/store/4cy2sqm7dd2rkg84kq5cm0g0njq18vzx-closure-info/registration console=ttyS0,115200n8 console=tty0270worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns271client # [ 0.002870] ACPI: Early table checksum verification disabled272worker2 # [ 0.000000] BIOS-provided physical RAM map:273lb # [ 0.002849] RAMDISK: [mem 0x5e36e000-0x5ffcffff]274worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor275client # [ 0.002874] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )276worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable277lb # [ 0.002857] ACPI: Early table checksum verification disabled278worker1 # [ 0.000811] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000279lb # [ 0.002861] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )280worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved281client # [ 0.002878] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)282worker1 # [ 0.000838] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs283worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved284lb # [ 0.002865] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)285worker1 # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT286worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable287client # [ 0.002883] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)288worker1 # [ 0.002730] found SMP MP-table at [mem 0x000f5450-0x000f545f]289lb # [ 0.002870] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)290worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved291worker1 # [ 0.002742] Using GB pages for direct mapping292lb # [ 0.002876] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)293client # [ 0.002890] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)294worker1 # [ 0.002880] RAMDISK: [mem 0x5e370000-0x5ffcffff]295worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved296lb # [ 0.002879] ACPI: FACS 0x000000005FFE0000 000040297client # [ 0.002893] ACPI: FACS 0x000000005FFE0000 000040298worker1 # [ 0.002888] ACPI: Early table checksum verification disabled299worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved300lb # [ 0.002880] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)301worker1 # [ 0.002891] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )302client # [ 0.002894] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)303worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved304lb # [ 0.002882] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)305worker1 # [ 0.002895] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)306worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved307client # [ 0.002896] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)308lb # [ 0.002884] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)309worker1 # [ 0.002900] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)310worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved311worker2 # [ 0.000000] NX (Execute Disable) protection: active312client # [ 0.002897] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)313lb # [ 0.002885] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)314worker1 # [ 0.002907] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)315worker2 # [ 0.000000] APIC: Static calls initialized316worker2 # [ 0.000000] SMBIOS 2.8 present.317worker1 # [ 0.002909] ACPI: FACS 0x000000005FFE0000 000040318lb # [ 0.002886] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]319client # [ 0.002899] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)320lb # [ 0.002888] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]321worker1 # [ 0.002911] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)322worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014323client # [ 0.002900] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]324worker2 # [ 0.000000] DMI: Memory slots populated: 1/1325lb # [ 0.002888] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]326worker1 # [ 0.002913] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)327worker2 # [ 0.000000] Hypervisor detected: KVM328client # [ 0.002901] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]329lb # [ 0.002889] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]330worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000331worker1 # [ 0.002915] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)332worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00333lb # [ 0.002889] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]334client # [ 0.002902] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]335worker1 # [ 0.002916] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)336worker2 # [ 0.000001] kvm-clock: using sched offset of 488442297 cycles337lb # [ 0.002889] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]338client # [ 0.002902] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]339worker1 # [ 0.002918] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]340lb # [ 0.002890] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]341worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns342client # [ 0.002903] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]343lb # [ 0.003117] No NUMA configuration found344worker1 # [ 0.002919] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]345worker2 # [ 0.000006] tsc: Detected 2400.012 MHz processor346lb # [ 0.003118] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]347worker2 # [ 0.000814] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000348worker1 # [ 0.002919] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]349client # [ 0.002903] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]350lb # [ 0.003121] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]351worker1 # [ 0.002920] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]352lb # [ 0.005408] Zone ranges:353worker2 # [ 0.000842] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs354client # [ 0.002904] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]355lb # [ 0.005409] DMA [mem 0x0000000000001000-0x0000000000ffffff]356client # [ 0.003136] No NUMA configuration found357worker1 # [ 0.002920] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]358worker2 # [ 0.000844] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT359lb # [ 0.005410] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]360worker2 # [ 0.002735] found SMP MP-table at [mem 0x000f5450-0x000f545f]361worker1 # [ 0.002921] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]362lb # [ 0.005411] Normal empty363client # [ 0.003137] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]364worker2 # [ 0.002747] Using GB pages for direct mapping365lb # [ 0.005412] Device empty366worker1 # [ 0.002921] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]367worker2 # [ 0.002901] RAMDISK: [mem 0x5e36f000-0x5ffcffff]368client # [ 0.003140] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]369lb # [ 0.005413] Movable zone start for each node370client # [ 0.005255] Zone ranges:371lb # [ 0.005414] Early memory node ranges372worker1 # [ 0.003140] No NUMA configuration found373worker2 # [ 0.002910] ACPI: Early table checksum verification disabled374worker1 # [ 0.003141] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]375lb # [ 0.005414] node 0: [mem 0x0000000000001000-0x000000000009efff]376worker2 # [ 0.002913] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )377client # [ 0.005256] DMA [mem 0x0000000000001000-0x0000000000ffffff]378lb # [ 0.005415] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]379worker1 # [ 0.003144] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]380worker1 # [ 0.005255] Zone ranges:381client # [ 0.005258] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]382worker2 # [ 0.002918] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)383client # [ 0.005259] Normal empty384lb # [ 0.005416] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]385worker1 # [ 0.005256] DMA [mem 0x0000000000001000-0x0000000000ffffff]386client # [ 0.005260] Device empty387worker2 # [ 0.002922] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)388lb # [ 0.005436] On node 0, zone DMA: 1 pages in unavailable ranges389client # [ 0.005260] Movable zone start for each node390worker1 # [ 0.005257] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]391worker1 # [ 0.005258] Normal empty392lb # [ 0.005699] On node 0, zone DMA: 97 pages in unavailable ranges393worker2 # [ 0.002930] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)394client # [ 0.005261] Early memory node ranges395worker1 # [ 0.005259] Device empty396worker2 # [ 0.002932] ACPI: FACS 0x000000005FFE0000 000040397lb # [ 0.031682] On node 0, zone DMA32: 40 pages in unavailable ranges398worker1 # [ 0.005260] Movable zone start for each node399client # [ 0.005261] node 0: [mem 0x0000000000001000-0x000000000009efff]400lb # [ 0.032155] ACPI: PM-Timer IO Port: 0x608401worker1 # [ 0.005260] Early memory node ranges402worker2 # [ 0.002933] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)403lb # [ 0.032168] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])404client # [ 0.005262] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]405worker1 # [ 0.005261] node 0: [mem 0x0000000000001000-0x000000000009efff]406lb # [ 0.032193] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23407worker2 # [ 0.002935] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)408worker1 # [ 0.005262] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]409client # [ 0.005263] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]410lb # [ 0.032196] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)411worker1 # [ 0.005263] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]412worker2 # [ 0.002937] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)413client # [ 0.005285] On node 0, zone DMA: 1 pages in unavailable ranges414lb # [ 0.032198] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)415worker1 # [ 0.005282] On node 0, zone DMA: 1 pages in unavailable ranges416client # [ 0.005534] On node 0, zone DMA: 97 pages in unavailable ranges417worker2 # [ 0.002938] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)418lb # [ 0.032199] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)419worker1 # [ 0.005683] On node 0, zone DMA: 97 pages in unavailable ranges420worker2 # [ 0.002939] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]421client # [ 0.031957] On node 0, zone DMA32: 40 pages in unavailable ranges422worker1 # [ 0.030842] On node 0, zone DMA32: 40 pages in unavailable ranges423lb # [ 0.032200] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)424client # [ 0.032432] ACPI: PM-Timer IO Port: 0x608425worker1 # [ 0.031297] ACPI: PM-Timer IO Port: 0x608426worker2 # [ 0.002940] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]427lb # [ 0.032201] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)428worker1 # [ 0.031310] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])429client # [ 0.032446] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])430worker2 # [ 0.002941] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]431lb # [ 0.032204] ACPI: Using ACPI (MADT) for SMP configuration information432worker1 # [ 0.031337] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23433lb # [ 0.032205] ACPI: HPET id: 0x8086a201 base: 0xfed00000434worker2 # [ 0.002941] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]435client # [ 0.032473] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23436lb # [ 0.032209] TSC deadline timer available437worker1 # [ 0.031340] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)438worker2 # [ 0.002942] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]439lb # [ 0.032213] CPU topo: Max. logical packages: 1440client # [ 0.032475] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)441worker1 # [ 0.031342] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)442lb # [ 0.032214] CPU topo: Max. logical dies: 1443worker2 # [ 0.002942] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]444client # [ 0.032477] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)445lb # [ 0.032214] CPU topo: Max. dies per package: 1446worker1 # [ 0.031343] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)447worker2 # [ 0.002943] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]448lb # [ 0.032217] CPU topo: Max. threads per core: 1449client # [ 0.032478] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)450worker2 # [ 0.003167] No NUMA configuration found451worker1 # [ 0.031344] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)452lb # [ 0.032218] CPU topo: Num. cores per package: 1453lb # [ 0.032218] CPU topo: Num. threads per package: 1454worker2 # [ 0.003168] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]455worker1 # [ 0.031345] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)456client # [ 0.032479] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)457worker2 # [ 0.003170] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]458lb # [ 0.032219] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs459worker1 # [ 0.031347] ACPI: Using ACPI (MADT) for SMP configuration information460worker2 # [ 0.005551] Zone ranges:461client # [ 0.032480] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)462worker1 # [ 0.031348] ACPI: HPET id: 0x8086a201 base: 0xfed00000463lb # [ 0.032242] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()464worker2 # [ 0.005553] DMA [mem 0x0000000000001000-0x0000000000ffffff]465worker1 # [ 0.031353] TSC deadline timer available466client # [ 0.032482] ACPI: Using ACPI (MADT) for SMP configuration information467worker1 # [ 0.031357] CPU topo: Max. logical packages: 1468worker2 # [ 0.005554] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]469lb # [ 0.032274] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]470worker2 # [ 0.005555] Normal empty471client # [ 0.032483] ACPI: HPET id: 0x8086a201 base: 0xfed00000472worker1 # [ 0.031357] CPU topo: Max. logical dies: 1473worker2 # [ 0.005556] Device empty474client # [ 0.032490] TSC deadline timer available475lb # [ 0.032275] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]476worker1 # [ 0.031358] CPU topo: Max. dies per package: 1477worker2 # [ 0.005557] Movable zone start for each node478worker2 # [ 0.005557] Early memory node ranges479client # [ 0.032494] CPU topo: Max. logical packages: 1480lb # [ 0.032277] [mem 0x60000000-0xafffffff] available for PCI devices481worker1 # [ 0.031361] CPU topo: Max. threads per core: 1482client # [ 0.032494] CPU topo: Max. logical dies: 1483lb # [ 0.032279] Booting paravirtualized kernel on KVM484worker1 # [ 0.031361] CPU topo: Num. cores per package: 1485worker2 # [ 0.005558] node 0: [mem 0x0000000000001000-0x000000000009efff]486client # [ 0.032495] CPU topo: Max. dies per package: 1487worker1 # [ 0.031362] CPU topo: Num. threads per package: 1488worker2 # [ 0.005559] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]489client # [ 0.032498] CPU topo: Max. threads per core: 1490lb # [ 0.032282] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns491worker1 # [ 0.031362] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs492client # [ 0.032499] CPU topo: Num. cores per package: 1493worker2 # [ 0.005560] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]494worker1 # [ 0.031384] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()495lb # [ 0.036761] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1496client # [ 0.032499] CPU topo: Num. threads per package: 1497worker2 # [ 0.005579] On node 0, zone DMA: 1 pages in unavailable ranges498lb # [ 0.037705] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152499client # [ 0.032499] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs500worker1 # [ 0.031417] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]501worker2 # [ 0.005836] On node 0, zone DMA: 97 pages in unavailable ranges502lb # [ 0.037757] kvm-guest: PV spinlocks disabled, single CPU503worker2 # [ 0.031362] On node 0, zone DMA32: 40 pages in unavailable ranges504client # [ 0.032520] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()505worker1 # [ 0.031418] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]506worker2 # [ 0.031840] ACPI: PM-Timer IO Port: 0x608507worker1 # [ 0.031420] [mem 0x60000000-0xafffffff] available for PCI devices508client # [ 0.032554] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]509worker2 # [ 0.031853] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])510worker1 # [ 0.031421] Booting paravirtualized kernel on KVM511client # [ 0.032556] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]512worker2 # [ 0.031881] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23513client # [ 0.032557] [mem 0x60000000-0xafffffff] available for PCI devices514worker1 # [ 0.031425] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns515worker2 # [ 0.031884] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)516client # [ 0.032559] Booting paravirtualized kernel on KVM517lb # [ 0.037759] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/ggsnp37phfgq4b11zirffk4vxgicg7h8-nixos-system-lb-test/init regInfo=/nix/store/94h72crvqzbw5sjsl7v7v31gd90khbjb-closure-info/registration console=ttyS0,115200n8 console=tty0518worker1 # [ 0.035904] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1519worker2 # [ 0.031886] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)520worker1 # [ 0.037990] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152521client # [ 0.032562] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns522worker2 # [ 0.031887] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)523worker1 # [ 0.038031] kvm-guest: PV spinlocks disabled, single CPU524lb # [ 0.037855] Unknown kernel command line parameters "regInfo=/nix/store/94h72crvqzbw5sjsl7v7v31gd90khbjb-closure-info/registration", will be passed to user space.525worker2 # [ 0.031888] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)526client # [ 0.037066] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1527lb # [ 0.037869] random: crng init done528worker2 # [ 0.031889] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)529client # [ 0.039111] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152530lb # [ 0.037870] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes531worker2 # [ 0.031891] ACPI: Using ACPI (MADT) for SMP configuration information532client # [ 0.039152] kvm-guest: PV spinlocks disabled, single CPU533worker2 # [ 0.031892] ACPI: HPET id: 0x8086a201 base: 0xfed00000534lb # [ 0.037961] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)535worker2 # [ 0.031896] TSC deadline timer available536lb # [ 0.038982] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)537worker2 # [ 0.031901] CPU topo: Max. logical packages: 1538lb # [ 0.039017] Fallback order for Node 0: 0539worker1 # [ 0.038033] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/1xcjwc0c3jblpxjbzzik759i2a779qhn-nixos-system-worker1-test/init regInfo=/nix/store/lplh0yfdzmjpfyfgf19r87dfbm8ls4wa-closure-info/registration console=ttyS0,115200n8 console=tty0540worker2 # [ 0.031901] CPU topo: Max. logical dies: 1541worker2 # [ 0.031902] CPU topo: Max. dies per package: 1542lb # [ 0.039020] Built 1 zonelists, mobility grouping on. Total pages: 393078543lb # [ 0.039021] Policy zone: DMA32544worker2 # [ 0.031905] CPU topo: Max. threads per core: 1545worker1 # [ 0.038128] Unknown kernel command line parameters "regInfo=/nix/store/lplh0yfdzmjpfyfgf19r87dfbm8ls4wa-closure-info/registration", will be passed to user space.546client # [ 0.039154] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/qgir8rpawilvsh39a93frcz2w73apwjr-nixos-system-client-test/init regInfo=/nix/store/lkz8fx5m8vmi31dy1j6w4dxp7rzq8kxy-closure-info/registration console=ttyS0,115200n8 console=tty0547worker2 # [ 0.031906] CPU topo: Num. cores per package: 1548lb # [ 0.039415] mem auto-init: stack:all(zero), heap alloc:on, heap free:off549worker1 # [ 0.038142] random: crng init done550worker2 # [ 0.031906] CPU topo: Num. threads per package: 1551lb # [ 0.042619] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1552worker1 # [ 0.038142] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes553worker2 # [ 0.031906] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs554lb # [ 0.045801] allocated 3145728 bytes of page_ext555client # [ 0.039248] Unknown kernel command line parameters "regInfo=/nix/store/lkz8fx5m8vmi31dy1j6w4dxp7rzq8kxy-closure-info/registration", will be passed to user space.556client # [ 0.039261] random: crng init done557worker1 # [ 0.040187] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)558worker2 # [ 0.031930] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()559lb # [ 0.055864] ftrace: allocating 48787 entries in 192 pages560lb # [ 0.055865] ftrace: allocated 192 pages with 2 groups561client # [ 0.039262] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes562worker1 # [ 0.041219] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)563lb # [ 0.056701] Dynamic Preempt: lazy564worker2 # [ 0.031965] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]565worker1 # [ 0.041255] Fallback order for Node 0: 0566client # [ 0.041450] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)567lb # [ 0.056988] rcu: Preemptible hierarchical RCU implementation.568worker2 # [ 0.031967] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]569worker1 # [ 0.041259] Built 1 zonelists, mobility grouping on. Total pages: 393078570lb # [ 0.056989] rcu: RCU event tracing is enabled.571worker1 # [ 0.041261] Policy zone: DMA32572client # [ 0.042473] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)573worker2 # [ 0.031968] [mem 0x60000000-0xafffffff] available for PCI devices574lb # [ 0.056990] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.575client # [ 0.042506] Fallback order for Node 0: 0576worker2 # [ 0.031970] Booting paravirtualized kernel on KVM577worker1 # [ 0.043991] mem auto-init: stack:all(zero), heap alloc:on, heap free:off578lb # [ 0.056991] Trampoline variant of Tasks RCU enabled.579client # [ 0.042509] Built 1 zonelists, mobility grouping on. Total pages: 393078580worker1 # [ 0.047278] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1581lb # [ 0.056991] Rude variant of Tasks RCU enabled.582client # [ 0.042511] Policy zone: DMA32583worker2 # [ 0.031974] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns584worker1 # [ 0.050421] allocated 3145728 bytes of page_ext585lb # [ 0.056992] Tracing variant of Tasks RCU enabled.586client # [ 0.044855] mem auto-init: stack:all(zero), heap alloc:on, heap free:off587worker1 # [ 0.060247] ftrace: allocating 48787 entries in 192 pages588worker2 # [ 0.036440] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1589lb # [ 0.056992] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.590client # [ 0.047916] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1591worker1 # [ 0.060249] ftrace: allocated 192 pages with 2 groups592worker2 # [ 0.038512] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152593worker1 # [ 0.061057] Dynamic Preempt: lazy594client # [ 0.051089] allocated 3145728 bytes of page_ext595lb # [ 0.056993] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1596worker2 # [ 0.038554] kvm-guest: PV spinlocks disabled, single CPU597worker1 # [ 0.061185] rcu: Preemptible hierarchical RCU implementation.598client # [ 0.061458] ftrace: allocating 48787 entries in 192 pages599lb # [ 0.057046] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.600worker1 # [ 0.061186] rcu: RCU event tracing is enabled.601client # [ 0.061461] ftrace: allocated 192 pages with 2 groups602client # [ 0.062362] Dynamic Preempt: lazy603worker1 # [ 0.061186] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.604lb # [ 0.057048] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.605client # [ 0.062482] rcu: Preemptible hierarchical RCU implementation.606worker1 # [ 0.061187] Trampoline variant of Tasks RCU enabled.607client # [ 0.062482] rcu: RCU event tracing is enabled.608worker1 # [ 0.061188] Rude variant of Tasks RCU enabled.609lb # [ 0.057049] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.610worker1 # [ 0.061188] Tracing variant of Tasks RCU enabled.611client # [ 0.062483] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.612lb # [ 0.061372] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16613worker2 # [ 0.038555] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/vjdbc1032z3354xwbcs1ba0jyv7imbfs-nixos-system-worker2-test/init regInfo=/nix/store/4cy2sqm7dd2rkg84kq5cm0g0njq18vzx-closure-info/registration console=ttyS0,115200n8 console=tty0614client # [ 0.062484] Trampoline variant of Tasks RCU enabled.615worker1 # [ 0.061189] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.616lb # [ 0.061636] rcu: srcu_init: Setting srcu_struct sizes based on contention.617client # [ 0.062485] Rude variant of Tasks RCU enabled.618worker1 # [ 0.061189] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1619client # [ 0.062485] Tracing variant of Tasks RCU enabled.620worker2 # [ 0.038658] Unknown kernel command line parameters "regInfo=/nix/store/4cy2sqm7dd2rkg84kq5cm0g0njq18vzx-closure-info/registration", will be passed to user space.621lb # [ 0.061642] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns622worker2 # [ 0.038671] random: crng init done623client # [ 0.062486] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.624worker1 # [ 0.061241] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.625worker2 # [ 0.038671] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes626client # [ 0.062487] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1627lb # [ 0.061757] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)628worker1 # [ 0.061243] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.629lb # [ 0.065266] Console: colour VGA+ 80x25630worker2 # [ 0.040715] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)631client # [ 0.062502] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.632lb # [ 0.065269] printk: legacy console [tty0] enabled633worker1 # [ 0.061244] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.634lb # [ 0.094697] printk: legacy console [ttyS0] enabled635worker2 # [ 0.041740] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)636worker1 # [ 0.065591] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16637client # [ 0.062504] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.638worker2 # [ 0.041772] Fallback order for Node 0: 0639lb # [ 0.191674] ACPI: Core revision 20250807640worker1 # [ 0.065860] rcu: srcu_init: Setting srcu_struct sizes based on contention.641worker2 # [ 0.041775] Built 1 zonelists, mobility grouping on. Total pages: 393078642client # [ 0.062505] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.643worker2 # [ 0.041776] Policy zone: DMA32644client # [ 0.067325] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16645lb # [ 0.192540] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns646worker1 # [ 0.065866] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns647worker2 # [ 0.044347] mem auto-init: stack:all(zero), heap alloc:on, heap free:off648client # [ 0.067632] rcu: srcu_init: Setting srcu_struct sizes based on contention.649lb # [ 0.194027] APIC: Switch to symmetric I/O mode setup650worker2 # [ 0.047500] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1651worker1 # [ 0.065970] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)652worker2 # [ 0.050707] allocated 3145728 bytes of page_ext653lb # [ 0.194958] x2apic enabled654worker1 # [ 0.069481] Console: colour VGA+ 80x25655client # [ 0.067637] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns656worker2 # [ 0.060494] ftrace: allocating 48787 entries in 192 pages657worker1 # [ 0.069484] printk: legacy console [tty0] enabled658lb # [ 0.195679] APIC: Switched APIC routing to: physical x2apic659worker2 # [ 0.060496] ftrace: allocated 192 pages with 2 groups660worker1 # [ 0.098900] printk: legacy console [ttyS0] enabled661worker2 # [ 0.061343] Dynamic Preempt: lazy662client # [ 0.067742] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)663worker1 # [ 0.196678] ACPI: Core revision 20250807664client # [ 0.071276] Console: colour VGA+ 80x25665worker2 # [ 0.061467] rcu: Preemptible hierarchical RCU implementation.666client # [ 0.071279] printk: legacy console [tty0] enabled667worker2 # [ 0.061468] rcu: RCU event tracing is enabled.668lb # [ 0.197470] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1669client # [ 0.100860] printk: legacy console [ttyS0] enabled670worker1 # [ 0.197506] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns671worker2 # [ 0.061469] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.672client # [ 0.208057] ACPI: Core revision 20250807673worker2 # [ 0.061470] Trampoline variant of Tasks RCU enabled.674worker1 # [ 0.198979] APIC: Switch to symmetric I/O mode setup675lb # [ 0.198442] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns676worker2 # [ 0.061471] Rude variant of Tasks RCU enabled.677worker2 # [ 0.061471] Tracing variant of Tasks RCU enabled.678worker1 # [ 0.199958] x2apic enabled679client # [ 0.208894] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns680lb # [ 0.200108] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)681worker2 # [ 0.061472] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.682worker1 # [ 0.200725] APIC: Switched APIC routing to: physical x2apic683client # [ 0.210391] APIC: Switch to symmetric I/O mode setup684worker2 # [ 0.061472] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1685lb # [ 0.201192] x86/cpu: User Mode Instruction Prevention (UMIP) activated686client # [ 0.211357] x2apic enabled687worker2 # [ 0.061487] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.688client # [ 0.212093] APIC: Switched APIC routing to: physical x2apic689worker1 # [ 0.202499] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1690lb # [ 0.202226] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127691worker2 # [ 0.061488] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.692lb # [ 0.203106] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0693worker1 # [ 0.203446] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns694client # [ 0.213831] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1695worker2 # [ 0.061489] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.696worker2 # [ 0.065912] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16697lb # [ 0.204110] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto698worker1 # [ 0.205079] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)699worker2 # [ 0.066193] rcu: srcu_init: Setting srcu_struct sizes based on contention.700client # [ 0.214806] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns701worker1 # [ 0.206161] x86/cpu: User Mode Instruction Prevention (UMIP) activated702lb # [ 0.206106] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl703worker2 # [ 0.066199] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns704client # [ 0.216433] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)705lb # [ 0.207106] Transient Scheduler Attacks: Vulnerable: No microcode706worker1 # [ 0.207191] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127707worker2 # [ 0.066302] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)708client # [ 0.217540] x86/cpu: User Mode Instruction Prevention (UMIP) activated709lb # [ 0.208105] Spectre V2 : Mitigation: Enhanced / Automatic IBRS710worker1 # [ 0.208077] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0711worker2 # [ 0.069821] Console: colour VGA+ 80x25712worker2 # [ 0.069823] printk: legacy console [tty0] enabled713client # [ 0.218570] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127714lb # [ 0.209106] Speculative Return Stack Overflow: Mitigation: Safe RET715worker2 # [ 0.099252] printk: legacy console [ttyS0] enabled716worker1 # [ 0.209080] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto717worker2 # [ 0.194839] ACPI: Core revision 20250807718client # [ 0.219430] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0719worker1 # [ 0.210077] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl720worker2 # [ 0.195719] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns721client # [ 0.220434] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto722worker1 # [ 0.211077] Transient Scheduler Attacks: Vulnerable: No microcode723worker2 # [ 0.197205] APIC: Switch to symmetric I/O mode setup724lb # [ 0.210105] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization725client # [ 0.221430] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl726worker1 # [ 0.212076] Spectre V2 : Mitigation: Enhanced / Automatic IBRS727worker2 # [ 0.198133] x2apic enabled728lb # [ 0.212113] Spectre V2 : Enabling IBPB for BPF729worker1 # [ 0.213077] Speculative Return Stack Overflow: Mitigation: Safe RET730client # [ 0.222430] Transient Scheduler Attacks: Vulnerable: No microcode731worker2 # [ 0.198865] APIC: Switched APIC routing to: physical x2apic732client # [ 0.223430] Spectre V2 : Mitigation: Enhanced / Automatic IBRS733lb # [ 0.213106] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier734lb # [ 0.214106] active return thunk: srso_alias_return_thunk735client # [ 0.224430] Speculative Return Stack Overflow: Mitigation: Safe RET736worker2 # [ 0.200672] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1737lb # [ 0.215745] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'738worker2 # [ 0.201625] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns739worker1 # [ 0.214076] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization740lb # [ 0.216106] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'741worker1 # [ 0.216083] Spectre V2 : Enabling IBPB for BPF742worker2 # [ 0.203259] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)743lb # [ 0.217868] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'744client # [ 0.225430] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization745worker1 # [ 0.217077] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier746worker2 # [ 0.204352] x86/cpu: User Mode Instruction Prevention (UMIP) activated747lb # [ 0.218885] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'748client # [ 0.227437] Spectre V2 : Enabling IBPB for BPF749worker1 # [ 0.218077] active return thunk: srso_alias_return_thunk750worker2 # [ 0.205379] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127751lb # [ 0.219833] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'752client # [ 0.228431] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier753worker1 # [ 0.219097] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'754lb # [ 0.220826] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'755worker2 # [ 0.206256] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0756client # [ 0.229431] active return thunk: srso_alias_return_thunk757worker1 # [ 0.220077] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'758lb # [ 0.221869] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'759worker2 # [ 0.207261] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto760worker1 # [ 0.221076] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'761client # [ 0.230452] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'762lb # [ 0.223063] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256763worker1 # [ 0.222077] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'764client # [ 0.231430] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'765worker2 # [ 0.208256] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl766lb # [ 0.224106] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64767worker1 # [ 0.223077] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'768worker2 # [ 0.209256] Transient Scheduler Attacks: Vulnerable: No microcode769client # [ 0.233154] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'770lb # [ 0.225106] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512771worker2 # [ 0.210256] Spectre V2 : Mitigation: Enhanced / Automatic IBRS772worker1 # [ 0.224076] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'773client # [ 0.234156] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'774lb # [ 0.226032] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024775worker2 # [ 0.211256] Speculative Return Stack Overflow: Mitigation: Safe RET776client # [ 0.234430] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'777worker1 # [ 0.225076] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'778lb # [ 0.226813] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8779client # [ 0.235430] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'780worker1 # [ 0.226077] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256781lb # [ 0.227813] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.782worker1 # [ 0.227076] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64783client # [ 0.236430] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'784worker1 # [ 0.228077] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512785client # [ 0.238430] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256786worker2 # [ 0.212256] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization787worker1 # [ 0.229076] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024788client # [ 0.239430] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64789worker2 # [ 0.214263] Spectre V2 : Enabling IBPB for BPF790worker1 # [ 0.230076] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8791client # [ 0.240430] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512792worker2 # [ 0.215257] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier793client # [ 0.241430] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024794worker1 # [ 0.231076] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.795worker2 # [ 0.216256] active return thunk: srso_alias_return_thunk796client # [ 0.242430] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8797worker2 # [ 0.217276] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'798client # [ 0.243431] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.799worker2 # [ 0.218256] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'800worker2 # [ 0.219256] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'801worker2 # [ 0.220256] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'802worker2 # [ 0.221256] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'803worker2 # [ 0.222256] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'804worker2 # [ 0.223256] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'805worker2 # [ 0.224257] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256806worker2 # [ 0.225971] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64807worker2 # [ 0.226256] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512808worker2 # [ 0.227256] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024809worker2 # [ 0.228256] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8810worker2 # [ 0.229256] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.811lb # [ 0.262256] Freeing SMP alternatives memory: 44K812lb # [ 0.263051] pid_max: default: 32768 minimum: 301813lb # [ 0.263794] LSM: initializing lsm=capability,landlock,yama,bpf,ima814lb # [ 0.264982] landlock: Up and running.815lb # [ 0.265567] Yama: becoming mindful.816lb # [ 0.266319] LSM support for eBPF active817worker1 # [ 0.266049] Freeing SMP alternatives memory: 44K818lb # [ 0.267222] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)819worker1 # [ 0.266676] pid_max: default: 32768 minimum: 301820worker1 # [ 0.267723] LSM: initializing lsm=capability,landlock,yama,bpf,ima821lb # [ 0.268140] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)822worker1 # [ 0.268185] landlock: Up and running.823client # [ 0.278412] Freeing SMP alternatives memory: 44K824worker1 # [ 0.269532] Yama: becoming mindful.825client # [ 0.279023] pid_max: default: 32768 minimum: 301826worker1 # [ 0.270290] LSM support for eBPF active827lb # [ 0.270972] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)828client # [ 0.280078] LSM: initializing lsm=capability,landlock,yama,bpf,ima829client # [ 0.280530] landlock: Up and running.830worker1 # [ 0.271176] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)831lb # [ 0.271619] Performance Events: Fam17h+ core perfctr, AMD PMU driver.832client # [ 0.281431] Yama: becoming mindful.833lb # [ 0.272110] ... version: 2834worker1 # [ 0.272111] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)835client # [ 0.282669] LSM support for eBPF active836lb # [ 0.272853] ... bit width: 48837lb # [ 0.273108] ... generic counters: 6838client # [ 0.283558] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)839lb # [ 0.273868] ... generic bitmap: 000000000000003f840worker1 # [ 0.274672] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)841client # [ 0.284472] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)842lb # [ 0.274152] ... fixed-purpose counters: 0843worker1 # [ 0.275611] Performance Events: Fam17h+ core perfctr, AMD PMU driver.844lb # [ 0.274879] ... fixed-purpose bitmap: 0000000000000000845worker1 # [ 0.276082] ... version: 2846lb # [ 0.275822] ... value mask: 0000ffffffffffff847client # [ 0.287300] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)848worker1 # [ 0.276839] ... bit width: 48849lb # [ 0.276108] ... max period: 00007fffffffffff850worker1 # [ 0.277079] ... generic counters: 6851lb # [ 0.277053] ... global_ctrl mask: 000000000000003f852client # [ 0.287939] Performance Events: Fam17h+ core perfctr, AMD PMU driver.853worker1 # [ 0.277836] ... generic bitmap: 000000000000003f854worker2 # [ 0.263311] Freeing SMP alternatives memory: 44K855lb # [ 0.277910] signal: max sigframe size: 3376856client # [ 0.288435] ... version: 2857worker1 # [ 0.278089] ... fixed-purpose counters: 0858worker2 # [ 0.264081] pid_max: default: 32768 minimum: 301859client # [ 0.289101] ... bit width: 48860lb # [ 0.278670] rcu: Hierarchical SRCU implementation.861worker1 # [ 0.278755] ... fixed-purpose bitmap: 0000000000000000862worker2 # [ 0.264882] LSM: initializing lsm=capability,landlock,yama,bpf,ima863client # [ 0.289432] ... generic counters: 6864lb # [ 0.279112] rcu: Max phase no-delay instances is 400.865worker1 # [ 0.279079] ... value mask: 0000ffffffffffff866worker2 # [ 0.265347] landlock: Up and running.867client # [ 0.290114] ... generic bitmap: 000000000000003f868worker2 # [ 0.266256] Yama: becoming mindful.869worker1 # [ 0.279941] ... max period: 00007fffffffffff870client # [ 0.290476] ... fixed-purpose counters: 0871worker2 # [ 0.267445] LSM support for eBPF active872worker1 # [ 0.280768] ... global_ctrl mask: 000000000000003f873client # [ 0.291187] ... fixed-purpose bitmap: 0000000000000000874worker1 # [ 0.281185] signal: max sigframe size: 3376875worker2 # [ 0.268212] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)876client # [ 0.292145] ... value mask: 0000ffffffffffff877worker1 # [ 0.281956] rcu: Hierarchical SRCU implementation.878client # [ 0.292433] ... max period: 00007fffffffffff879worker2 # [ 0.269297] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)880worker1 # [ 0.282656] rcu: Max phase no-delay instances is 400.881client # [ 0.293329] ... global_ctrl mask: 000000000000003f882lb # [ 0.284480] smp: Bringing up secondary CPUs ...883client # [ 0.294266] signal: max sigframe size: 3376884lb # [ 0.285122] smp: Brought up 1 node, 1 CPU885client # [ 0.295048] rcu: Hierarchical SRCU implementation.886worker2 # [ 0.271671] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)887lb # [ 0.285814] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)888client # [ 0.295437] rcu: Max phase no-delay instances is 400.889worker2 # [ 0.272796] Performance Events: Fam17h+ core perfctr, AMD PMU driver.890worker2 # [ 0.273266] ... version: 2891lb # [ 0.286263] Memory: 1454608K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109316K reserved, 0K cma-reserved)892worker1 # [ 0.287639] smp: Bringing up secondary CPUs ...893worker2 # [ 0.273984] ... bit width: 48894lb # [ 0.287368] devtmpfs: initialized895worker1 # [ 0.288093] smp: Brought up 1 node, 1 CPU896worker2 # [ 0.274258] ... generic counters: 6897lb # [ 0.288162] x86/mm: Memory block size: 128MB898worker2 # [ 0.274982] ... generic bitmap: 000000000000003f899worker1 # [ 0.288780] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)900worker2 # [ 0.275258] ... fixed-purpose counters: 0901client # [ 0.300950] smp: Bringing up secondary CPUs ...902worker2 # [ 0.276021] ... fixed-purpose bitmap: 0000000000000000903lb # [ 0.289912] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)904client # [ 0.301446] smp: Brought up 1 node, 1 CPU905worker1 # [ 0.289283] Memory: 1454608K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109308K reserved, 0K cma-reserved)906worker2 # [ 0.276258] ... value mask: 0000ffffffffffff907worker1 # [ 0.290295] devtmpfs: initialized908client # [ 0.302129] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)909lb # [ 0.291009] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).910worker2 # [ 0.277214] ... max period: 00007fffffffffff911worker1 # [ 0.291140] x86/mm: Memory block size: 128MB912lb # [ 0.292177] pinctrl core: initialized pinctrl subsystem913worker2 # [ 0.277917] ... global_ctrl mask: 000000000000003f914worker2 # [ 0.278355] signal: max sigframe size: 3376915lb # [ 0.293347] PM: RTC time: 05:11:49, date: 2026-09-23916client # [ 0.302580] Memory: 1454596K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109308K reserved, 0K cma-reserved)917worker2 # [ 0.279154] rcu: Hierarchical SRCU implementation.918worker1 # [ 0.292851] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)919client # [ 0.303714] devtmpfs: initialized920worker2 # [ 0.279858] rcu: Max phase no-delay instances is 400.921client # [ 0.304558] x86/mm: Memory block size: 128MB922worker1 # [ 0.293987] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).923worker1 # [ 0.295167] pinctrl core: initialized pinctrl subsystem924lb # [ 0.296714] NET: Registered PF_NETLINK/PF_ROUTE protocol family925client # [ 0.306311] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)926worker1 # [ 0.296355] PM: RTC time: 05:11:49, date: 2026-09-23927lb # [ 0.297575] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations928client # [ 0.307381] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).929lb # [ 0.298128] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations930client # [ 0.308511] pinctrl core: initialized pinctrl subsystem931worker2 # [ 0.284931] smp: Bringing up secondary CPUs ...932worker2 # [ 0.285271] smp: Brought up 1 node, 1 CPU933client # [ 0.309639] PM: RTC time: 05:11:49, date: 2026-09-23934lb # [ 0.299367] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations935worker2 # [ 0.285919] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)936worker1 # [ 0.300343] NET: Registered PF_NETLINK/PF_ROUTE protocol family937lb # [ 0.300118] audit: initializing netlink subsys (disabled)938lb # [ 0.301302] thermal_sys: Registered thermal governor 'fair_share'939worker1 # [ 0.301603] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations940worker2 # [ 0.286409] Memory: 1454604K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109312K reserved, 0K cma-reserved)941lb # [ 0.301304] thermal_sys: Registered thermal governor 'bang_bang'942client # [ 0.312994] NET: Registered PF_NETLINK/PF_ROUTE protocol family943worker2 # [ 0.287492] devtmpfs: initialized944worker1 # [ 0.302106] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations945worker2 # [ 0.288270] x86/mm: Memory block size: 128MB946lb # [ 0.302112] audit: type=2000 audit(1790140310.246:1): state=initialized audit_enabled=0 res=1947client # [ 0.313921] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations948worker1 # [ 0.303382] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations949lb # [ 0.304110] thermal_sys: Registered thermal governor 'step_wise'950client # [ 0.314460] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations951worker1 # [ 0.304089] audit: initializing netlink subsys (disabled)952lb # [ 0.304112] thermal_sys: Registered thermal governor 'user_space'953worker2 # [ 0.289985] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)954worker1 # [ 0.305207] thermal_sys: Registered thermal governor 'fair_share'955lb # [ 0.305101] thermal_sys: Registered thermal governor 'power_allocator'956client # [ 0.315691] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations957worker2 # [ 0.291162] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).958lb # [ 0.305844] cpuidle: using governor menu959worker1 # [ 0.305210] thermal_sys: Registered thermal governor 'bang_bang'960client # [ 0.316444] audit: initializing netlink subsys (disabled)961worker2 # [ 0.292360] pinctrl core: initialized pinctrl subsystem962worker1 # [ 0.306080] thermal_sys: Registered thermal governor 'step_wise'963lb # [ 0.307316] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5964client # [ 0.317435] audit: type=2000 audit(1790140310.242:1): state=initialized audit_enabled=0 res=1965worker1 # [ 0.306996] thermal_sys: Registered thermal governor 'user_space'966worker2 # [ 0.293466] PM: RTC time: 05:11:49, date: 2026-09-23967client # [ 0.318566] thermal_sys: Registered thermal governor 'fair_share'968lb # [ 0.308338] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]969worker1 # [ 0.307805] audit: type=2000 audit(1790140310.248:1): state=initialized audit_enabled=0 res=1970client # [ 0.318568] thermal_sys: Registered thermal governor 'bang_bang'971lb # [ 0.309111] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry972worker1 # [ 0.309082] thermal_sys: Registered thermal governor 'power_allocator'973client # [ 0.319433] thermal_sys: Registered thermal governor 'step_wise'974worker1 # [ 0.309098] cpuidle: using governor menu975lb # [ 0.310120] PCI: Using configuration type 1 for base access976worker2 # [ 0.296899] NET: Registered PF_NETLINK/PF_ROUTE protocol family977client # [ 0.320422] thermal_sys: Registered thermal governor 'user_space'978worker1 # [ 0.311278] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5979lb # [ 0.311175] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.980client # [ 0.321174] thermal_sys: Registered thermal governor 'power_allocator'981worker2 # [ 0.297738] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations982client # [ 0.321454] cpuidle: using governor menu983worker1 # [ 0.312320] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]984worker2 # [ 0.298279] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations985client # [ 0.323657] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5986worker1 # [ 0.313082] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry987worker2 # [ 0.299522] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations988worker1 # [ 0.314091] PCI: Using configuration type 1 for base access989worker2 # [ 0.300269] audit: initializing netlink subsys (disabled)990client # [ 0.324683] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]991worker1 # [ 0.315234] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.992worker2 # [ 0.301396] thermal_sys: Registered thermal governor 'fair_share'993client # [ 0.325436] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry994worker2 # [ 0.301399] thermal_sys: Registered thermal governor 'bang_bang'995client # [ 0.326445] PCI: Using configuration type 1 for base access996lb # [ 0.318123] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages997worker2 # [ 0.302259] thermal_sys: Registered thermal governor 'step_wise'998client # [ 0.327565] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.999lb # [ 0.319108] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1000worker2 # [ 0.303225] thermal_sys: Registered thermal governor 'user_space'1001worker2 # [ 0.304018] audit: type=2000 audit(1790140310.263:1): state=initialized audit_enabled=0 res=11002worker2 # [ 0.305262] thermal_sys: Registered thermal governor 'power_allocator'1003worker2 # [ 0.305301] cpuidle: using governor menu1004worker1 # [ 0.322378] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1005worker2 # [ 0.307517] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.51006worker1 # [ 0.323080] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1007lb # [ 0.324110] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1008worker2 # [ 0.308553] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]1009client # [ 0.334453] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1010lb # [ 0.325124] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1011worker2 # [ 0.309262] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry1012client # [ 0.335433] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1013worker2 # [ 0.310270] PCI: Using configuration type 1 for base access1014worker2 # [ 0.311381] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.1015worker1 # [ 0.328081] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1016worker1 # [ 0.329081] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1017client # [ 0.340435] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1018client # [ 0.341434] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1019worker2 # [ 0.316279] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1020worker2 # [ 0.317259] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1021lb # [ 0.336340] ACPI: Added _OSI(Module Device)1022lb # [ 0.337058] ACPI: Added _OSI(Processor Device)1023lb # [ 0.337657] ACPI: Added _OSI(Processor Aggregator Device)1024worker1 # [ 0.337810] ACPI: Added _OSI(Module Device)1025worker1 # [ 0.338082] ACPI: Added _OSI(Processor Device)1026worker1 # [ 0.338801] ACPI: Added _OSI(Processor Aggregator Device)1027worker2 # [ 0.322259] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1028worker2 # [ 0.323259] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1029client # [ 0.352720] ACPI: Added _OSI(Module Device)1030client # [ 0.353413] ACPI: Added _OSI(Processor Device)1031client # [ 0.353986] ACPI: Added _OSI(Processor Aggregator Device)1032lb # [ 0.346246] ACPI: 1 ACPI AML tables successfully acquired and loaded1033worker1 # [ 0.345734] ACPI: 1 ACPI AML tables successfully acquired and loaded1034lb # [ 0.348047] ACPI: Interpreter enabled1035worker2 # [ 0.331281] ACPI: Added _OSI(Module Device)1036lb # [ 0.348562] ACPI: PM: (supports S0 S3 S4 S5)1037worker2 # [ 0.332019] ACPI: Added _OSI(Processor Device)1038lb # [ 0.349109] ACPI: Using IOAPIC for interrupt routing1039worker2 # [ 0.332260] ACPI: Added _OSI(Processor Aggregator Device)1040worker1 # [ 0.349146] ACPI: Interpreter enabled1041lb # [ 0.349953] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1042worker1 # [ 0.349910] ACPI: PM: (supports S0 S3 S4 S5)1043worker1 # [ 0.350587] ACPI: Using IOAPIC for interrupt routing1044client # [ 0.362567] ACPI: 1 ACPI AML tables successfully acquired and loaded1045lb # [ 0.355110] PCI: Using E820 reservations for host bridge windows1046worker1 # [ 0.353152] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1047lb # [ 0.356172] ACPI: Enabled 2 GPEs in block 00 to 3F1048worker1 # [ 0.354081] PCI: Using E820 reservations for host bridge windows1049client # [ 0.366756] ACPI: Interpreter enabled1050client # [ 0.367451] ACPI: PM: (supports S0 S3 S4 S5)1051worker2 # [ 0.341178] ACPI: 1 ACPI AML tables successfully acquired and loaded1052client # [ 0.368138] ACPI: Using IOAPIC for interrupt routing1053worker1 # [ 0.357219] ACPI: Enabled 2 GPEs in block 00 to 3F1054client # [ 0.368480] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1055client # [ 0.369434] PCI: Using E820 reservations for host bridge windows1056client # [ 0.372514] ACPI: Enabled 2 GPEs in block 00 to 3F1057worker2 # [ 0.347614] ACPI: Interpreter enabled1058worker2 # [ 0.348274] ACPI: PM: (supports S0 S3 S4 S5)1059lb # [ 0.365044] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1060worker2 # [ 0.348972] ACPI: Using IOAPIC for interrupt routing1061lb # [ 0.365944] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1062worker2 # [ 0.349336] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1063lb # [ 0.367185] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1064worker1 # [ 0.365945] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1065worker2 # [ 0.350260] PCI: Using E820 reservations for host bridge windows1066lb # [ 0.368230] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1067worker1 # [ 0.366879] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1068lb # [ 0.369470] PCI host bridge to bus 0000:001069lb # [ 0.370113] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1070worker2 # [ 0.353318] ACPI: Enabled 2 GPEs in block 00 to 3F1071lb # [ 0.371108] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1072worker1 # [ 0.369180] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1073client # [ 0.381321] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1074lb # [ 0.372109] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1075worker1 # [ 0.370212] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1076client # [ 0.382272] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1077lb # [ 0.373111] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1078worker1 # [ 0.371487] PCI host bridge to bus 0000:001079client # [ 0.383532] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1080worker1 # [ 0.372088] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1081lb # [ 0.374111] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1082client # [ 0.384591] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1083worker1 # [ 0.373082] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1084lb # [ 0.375111] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1085client # [ 0.385835] PCI host bridge to bus 0000:001086lb # [ 0.376111] pci_bus 0000:00: root bus resource [bus 00-ff]1087worker1 # [ 0.374082] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1088client # [ 0.386441] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1089worker1 # [ 0.375082] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1090worker2 # [ 0.361937] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1091lb # [ 0.377224] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1092client # [ 0.387435] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1093worker1 # [ 0.376081] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1094worker2 # [ 0.362265] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1095client # [ 0.388435] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1096lb # [ 0.378619] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1097worker1 # [ 0.377081] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1098client # [ 0.389435] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1099worker1 # [ 0.378082] pci_bus 0000:00: root bus resource [bus 00-ff]1100worker2 # [ 0.364452] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1101client # [ 0.390435] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1102worker1 # [ 0.379072] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1103worker2 # [ 0.365382] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1104client # [ 0.391434] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1105worker2 # [ 0.366644] PCI host bridge to bus 0000:001106worker1 # [ 0.380526] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1107client # [ 0.392436] pci_bus 0000:00: root bus resource [bus 00-ff]1108worker2 # [ 0.367263] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1109client # [ 0.393511] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1110worker2 # [ 0.368259] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1111worker2 # [ 0.369259] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1112client # [ 0.394875] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1113worker2 # [ 0.370259] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1114lb # [ 0.381233] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1115worker2 # [ 0.371259] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1116lb # [ 0.382254] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1117worker1 # [ 0.383152] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1118lb # [ 0.383142] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1119worker2 # [ 0.372273] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1120worker2 # [ 0.373262] pci_bus 0000:00: root bus resource [bus 00-ff]1121worker1 # [ 0.384093] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1122lb # [ 0.384522] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1123worker2 # [ 0.374357] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1124worker1 # [ 0.385223] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1125worker2 # [ 0.375733] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1126worker1 # [ 0.386438] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1127lb # [ 0.386051] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1128client # [ 0.397560] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1129client # [ 0.398449] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1130worker1 # [ 0.387857] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1131client # [ 0.399485] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1132lb # [ 0.389129] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1133client # [ 0.400822] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1134lb # [ 0.389960] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1135worker1 # [ 0.391098] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1136lb # [ 0.391131] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1137worker1 # [ 0.391983] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1138client # [ 0.402240] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1139lb # [ 0.392115] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1140worker1 # [ 0.392822] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1141worker2 # [ 0.378348] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1142worker2 # [ 0.379271] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1143lb # [ 0.393627] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1144worker1 # [ 0.393085] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1145worker2 # [ 0.380213] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1146client # [ 0.405451] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1147client # [ 0.406292] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1148worker1 # [ 0.394624] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1149worker2 # [ 0.381406] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1150lb # [ 0.396030] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1151client # [ 0.407158] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1152lb # [ 0.396726] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1153client # [ 0.407440] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1154worker2 # [ 0.382997] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1155lb # [ 0.397140] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1156worker1 # [ 0.396111] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1157worker1 # [ 0.396941] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1158client # [ 0.409025] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1159worker1 # [ 0.397826] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1160lb # [ 0.398677] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1161worker2 # [ 0.385286] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1162worker2 # [ 0.386119] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1163client # [ 0.410456] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1164worker1 # [ 0.399485] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1165worker2 # [ 0.386988] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1166client # [ 0.411282] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1167lb # [ 0.401109] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1168worker2 # [ 0.387265] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1169client # [ 0.412139] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1170lb # [ 0.401954] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1171lb # [ 0.402840] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1172worker2 # [ 0.388801] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1173client # [ 0.412996] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1174worker1 # [ 0.402088] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1175worker1 # [ 0.402920] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1176lb # [ 0.403653] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1177worker1 # [ 0.403861] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1178worker2 # [ 0.390268] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1179worker2 # [ 0.391107] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1180client # [ 0.415454] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1181worker1 # [ 0.405508] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1182worker2 # [ 0.391992] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1183client # [ 0.416304] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1184client # [ 0.417149] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1185lb # [ 0.406944] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1186worker2 # [ 0.392830] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1187lb # [ 0.407745] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1188worker1 # [ 0.407945] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1189client # [ 0.418006] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1190lb # [ 0.408131] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1191worker1 # [ 0.408721] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1192lb # [ 0.409115] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1193worker1 # [ 0.409101] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1194client # [ 0.419459] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1195lb # [ 0.410678] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1196worker1 # [ 0.410086] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1197worker2 # [ 0.395275] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1198client # [ 0.420297] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1199worker2 # [ 0.396265] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1200worker1 # [ 0.411587] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1201lb # [ 0.412122] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1202client # [ 0.421165] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1203worker2 # [ 0.397205] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1204lb # [ 0.413059] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1205client # [ 0.421440] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1206worker1 # [ 0.413123] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1207worker2 # [ 0.398700] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1208worker1 # [ 0.414095] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1209client # [ 0.422963] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1210lb # [ 0.414516] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1211worker2 # [ 0.400266] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1212worker1 # [ 0.415652] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1213client # [ 0.424446] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1214worker2 # [ 0.401105] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1215lb # [ 0.416122] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1216client # [ 0.425376] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1217worker2 # [ 0.401968] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1218lb # [ 0.417070] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1219worker2 # [ 0.402265] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1220worker1 # [ 0.417092] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1221client # [ 0.426814] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1222lb # [ 0.418535] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1223worker1 # [ 0.418102] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1224worker2 # [ 0.404226] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1225client # [ 0.428459] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1226worker1 # [ 0.419663] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1227lb # [ 0.420122] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1228worker2 # [ 0.406122] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1229client # [ 0.429456] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1230lb # [ 0.421108] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1231worker2 # [ 0.406977] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1232worker1 # [ 0.421092] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1233client # [ 0.431063] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1234worker1 # [ 0.422102] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1235worker2 # [ 0.407893] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1236lb # [ 0.423221] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1237client # [ 0.432446] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1238worker1 # [ 0.423652] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1239worker2 # [ 0.409299] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1240lb # [ 0.425131] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1241client # [ 0.433405] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1242worker2 # [ 0.410233] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1243lb # [ 0.426073] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1244client # [ 0.434810] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1245worker1 # [ 0.425934] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1246worker2 # [ 0.411973] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1247lb # [ 0.427505] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1248worker1 # [ 0.426837] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1249client # [ 0.436446] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1250worker2 # [ 0.414090] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1251worker1 # [ 0.427836] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1252client # [ 0.437461] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1253lb # [ 0.429130] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1254worker2 # [ 0.415019] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1255client # [ 0.438990] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1256lb # [ 0.429964] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1257worker2 # [ 0.415827] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1258worker1 # [ 0.430782] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1259lb # [ 0.431132] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1260worker1 # [ 0.431088] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1261worker2 # [ 0.417259] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1262client # [ 0.440459] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1263worker1 # [ 0.432032] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1264lb # [ 0.432665] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1265client # [ 0.441294] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1266worker2 # [ 0.418225] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1267client # [ 0.442154] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1268worker1 # [ 0.433459] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1269lb # [ 0.434800] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1270worker2 # [ 0.419699] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1271client # [ 0.442986] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1272lb # [ 0.435116] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1273worker1 # [ 0.435113] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1274lb # [ 0.436059] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1275worker2 # [ 0.422275] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1276worker1 # [ 0.435959] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1277worker2 # [ 0.423147] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1278client # [ 0.445170] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1279worker1 # [ 0.436784] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1280lb # [ 0.437494] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1281worker2 # [ 0.424018] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1282client # [ 0.445454] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1283client # [ 0.446369] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1284worker1 # [ 0.437649] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1285worker2 # [ 0.424797] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1286lb # [ 0.439589] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1287worker1 # [ 0.438908] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1288client # [ 0.447803] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1289lb # [ 0.440340] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1290worker2 # [ 0.426267] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1291client # [ 0.449083] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1292worker1 # [ 0.439938] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1293lb # [ 0.441844] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1294worker2 # [ 0.427112] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1295client # [ 0.449638] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1296worker1 # [ 0.441636] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1297worker2 # [ 0.428280] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1298lb # [ 0.442319] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1299worker1 # [ 0.442290] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1300client # [ 0.451258] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1301lb # [ 0.443831] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1302worker2 # [ 0.429816] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1303worker1 # [ 0.443641] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1304client # [ 0.452677] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1305worker2 # [ 0.430793] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1306worker2 # [ 0.431462] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1307client # [ 0.454420] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1308worker2 # [ 0.432830] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1309worker2 # [ 0.433452] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1310worker2 # [ 0.434795] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1311worker1 # [ 0.444337] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1312lb # [ 0.444336] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1313worker1 # [ 0.446655] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1314worker1 # [ 0.447365] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1315lb # [ 0.447810] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1316worker1 # [ 0.448488] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1317client # [ 0.455477] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1318lb # [ 0.448387] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1319worker1 # [ 0.449352] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1320client # [ 0.458082] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1321lb # [ 0.449392] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1322worker2 # [ 0.435529] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1323client # [ 0.458889] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1324worker1 # [ 0.451111] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1325lb # [ 0.450389] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1326worker1 # [ 0.451955] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1327worker2 # [ 0.438260] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1328client # [ 0.459740] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1329worker1 # [ 0.453129] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1330worker2 # [ 0.439451] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1331client # [ 0.460694] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1332lb # [ 0.452157] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1333worker1 # [ 0.454791] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1334lb # [ 0.453002] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1335worker2 # [ 0.440547] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1336client # [ 0.462455] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1337client # [ 0.463288] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1338worker2 # [ 0.441521] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1339lb # [ 0.454354] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1340client # [ 0.464482] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1341lb # [ 0.455945] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1342worker2 # [ 0.443309] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1343worker1 # [ 0.458745] ACPI: PCI: Interrupt link LNKA configured for IRQ 101344client # [ 0.466057] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1345worker2 # [ 0.444158] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1346worker1 # [ 0.459196] ACPI: PCI: Interrupt link LNKB configured for IRQ 101347worker1 # [ 0.460181] ACPI: PCI: Interrupt link LNKC configured for IRQ 111348worker2 # [ 0.445309] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1349worker1 # [ 0.461191] ACPI: PCI: Interrupt link LNKD configured for IRQ 111350lb # [ 0.459796] ACPI: PCI: Interrupt link LNKA configured for IRQ 101351worker1 # [ 0.462185] ACPI: PCI: Interrupt link LNKE configured for IRQ 101352worker2 # [ 0.446891] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1353lb # [ 0.460211] ACPI: PCI: Interrupt link LNKB configured for IRQ 101354client # [ 0.469523] ACPI: PCI: Interrupt link LNKA configured for IRQ 101355worker1 # [ 0.463187] ACPI: PCI: Interrupt link LNKF configured for IRQ 101356lb # [ 0.461205] ACPI: PCI: Interrupt link LNKC configured for IRQ 111357client # [ 0.470544] ACPI: PCI: Interrupt link LNKB configured for IRQ 101358worker1 # [ 0.464190] ACPI: PCI: Interrupt link LNKG configured for IRQ 111359lb # [ 0.462208] ACPI: PCI: Interrupt link LNKD configured for IRQ 111360client # [ 0.471539] ACPI: PCI: Interrupt link LNKC configured for IRQ 111361lb # [ 0.463219] ACPI: PCI: Interrupt link LNKE configured for IRQ 101362worker1 # [ 0.465211] ACPI: PCI: Interrupt link LNKH configured for IRQ 111363client # [ 0.472534] ACPI: PCI: Interrupt link LNKD configured for IRQ 111364worker2 # [ 0.450292] ACPI: PCI: Interrupt link LNKA configured for IRQ 101365lb # [ 0.464208] ACPI: PCI: Interrupt link LNKF configured for IRQ 101366worker1 # [ 0.466139] ACPI: PCI: Interrupt link GSIA configured for IRQ 161367client # [ 0.473537] ACPI: PCI: Interrupt link LNKE configured for IRQ 101368worker2 # [ 0.451339] ACPI: PCI: Interrupt link LNKB configured for IRQ 101369lb # [ 0.465213] ACPI: PCI: Interrupt link LNKG configured for IRQ 111370worker1 # [ 0.467114] ACPI: PCI: Interrupt link GSIB configured for IRQ 171371client # [ 0.474537] ACPI: PCI: Interrupt link LNKF configured for IRQ 101372worker2 # [ 0.452356] ACPI: PCI: Interrupt link LNKC configured for IRQ 111373lb # [ 0.466213] ACPI: PCI: Interrupt link LNKH configured for IRQ 111374worker1 # [ 0.468110] ACPI: PCI: Interrupt link GSIC configured for IRQ 181375worker2 # [ 0.453357] ACPI: PCI: Interrupt link LNKD configured for IRQ 111376client # [ 0.475548] ACPI: PCI: Interrupt link LNKG configured for IRQ 111377lb # [ 0.467148] ACPI: PCI: Interrupt link GSIA configured for IRQ 161378worker1 # [ 0.469070] ACPI: PCI: Interrupt link GSID configured for IRQ 191379client # [ 0.476551] ACPI: PCI: Interrupt link LNKH configured for IRQ 111380worker2 # [ 0.454358] ACPI: PCI: Interrupt link LNKE configured for IRQ 101381lb # [ 0.468086] ACPI: PCI: Interrupt link GSIB configured for IRQ 171382worker1 # [ 0.469798] ACPI: PCI: Interrupt link GSIE configured for IRQ 201383client # [ 0.477481] ACPI: PCI: Interrupt link GSIA configured for IRQ 161384worker2 # [ 0.455355] ACPI: PCI: Interrupt link LNKF configured for IRQ 101385lb # [ 0.468831] ACPI: PCI: Interrupt link GSIC configured for IRQ 181386worker1 # [ 0.470096] ACPI: PCI: Interrupt link GSIF configured for IRQ 211387client # [ 0.478405] ACPI: PCI: Interrupt link GSIB configured for IRQ 171388worker2 # [ 0.456359] ACPI: PCI: Interrupt link LNKG configured for IRQ 111389lb # [ 0.469125] ACPI: PCI: Interrupt link GSID configured for IRQ 191390worker1 # [ 0.471029] ACPI: PCI: Interrupt link GSIG configured for IRQ 221391client # [ 0.479137] ACPI: PCI: Interrupt link GSIC configured for IRQ 181392worker2 # [ 0.457356] ACPI: PCI: Interrupt link LNKH configured for IRQ 111393lb # [ 0.470070] ACPI: PCI: Interrupt link GSIE configured for IRQ 201394worker1 # [ 0.471791] ACPI: PCI: Interrupt link GSIH configured for IRQ 231395client # [ 0.479458] ACPI: PCI: Interrupt link GSID configured for IRQ 191396worker2 # [ 0.458295] ACPI: PCI: Interrupt link GSIA configured for IRQ 161397lb # [ 0.470832] ACPI: PCI: Interrupt link GSIF configured for IRQ 211398client # [ 0.480376] ACPI: PCI: Interrupt link GSIE configured for IRQ 201399worker2 # [ 0.459228] ACPI: PCI: Interrupt link GSIB configured for IRQ 171400lb # [ 0.471123] ACPI: PCI: Interrupt link GSIG configured for IRQ 221401worker1 # [ 0.473332] iommu: Default domain type: Translated1402client # [ 0.481193] ACPI: PCI: Interrupt link GSIF configured for IRQ 211403worker2 # [ 0.459993] ACPI: PCI: Interrupt link GSIC configured for IRQ 181404lb # [ 0.472056] ACPI: PCI: Interrupt link GSIH configured for IRQ 231405worker1 # [ 0.474087] iommu: DMA domain TLB invalidation policy: lazy mode1406client # [ 0.481446] ACPI: PCI: Interrupt link GSIG configured for IRQ 221407worker2 # [ 0.460271] ACPI: PCI: Interrupt link GSID configured for IRQ 191408worker1 # [ 0.475358] ACPI: bus type USB registered1409client # [ 0.482405] ACPI: PCI: Interrupt link GSIH configured for IRQ 231410worker2 # [ 0.461249] ACPI: PCI: Interrupt link GSIE configured for IRQ 201411lb # [ 0.473832] iommu: Default domain type: Translated1412worker1 # [ 0.476103] usbcore: registered new interface driver usbfs1413worker2 # [ 0.461996] ACPI: PCI: Interrupt link GSIF configured for IRQ 211414lb # [ 0.474116] iommu: DMA domain TLB invalidation policy: lazy mode1415worker1 # [ 0.476967] usbcore: registered new interface driver hub1416client # [ 0.484224] iommu: Default domain type: Translated1417lb # [ 0.475314] ACPI: bus type USB registered1418worker2 # [ 0.462273] ACPI: PCI: Interrupt link GSIG configured for IRQ 221419worker1 # [ 0.477701] usbcore: registered new device driver usb1420client # [ 0.485029] iommu: DMA domain TLB invalidation policy: lazy mode1421lb # [ 0.476025] usbcore: registered new interface driver usbfs1422worker2 # [ 0.463273] ACPI: PCI: Interrupt link GSIH configured for IRQ 231423client # [ 0.485704] ACPI: bus type USB registered1424worker1 # [ 0.479117] NetLabel: Initializing1425lb # [ 0.476746] usbcore: registered new interface driver hub1426worker1 # [ 0.479721] NetLabel: domain hash size = 1281427client # [ 0.486463] usbcore: registered new interface driver usbfs1428lb # [ 0.477128] usbcore: registered new device driver usb1429worker2 # [ 0.465316] iommu: Default domain type: Translated1430worker1 # [ 0.480081] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1431client # [ 0.487342] usbcore: registered new interface driver hub1432worker2 # [ 0.466094] iommu: DMA domain TLB invalidation policy: lazy mode1433lb # [ 0.478880] NetLabel: Initializing1434worker1 # [ 0.480989] NetLabel: unlabeled traffic allowed by default1435client # [ 0.488087] usbcore: registered new device driver usb1436worker2 # [ 0.467286] ACPI: bus type USB registered1437worker1 # [ 0.481728] PCI: Using ACPI for IRQ routing1438lb # [ 0.479115] NetLabel: domain hash size = 1281439worker2 # [ 0.468049] usbcore: registered new interface driver usbfs1440client # [ 0.489476] NetLabel: Initializing1441lb # [ 0.479989] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1442client # [ 0.490065] NetLabel: domain hash size = 1281443worker2 # [ 0.468941] usbcore: registered new interface driver hub1444lb # [ 0.480853] NetLabel: unlabeled traffic allowed by default1445worker2 # [ 0.469267] usbcore: registered new device driver usb1446client # [ 0.490433] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1447lb # [ 0.481121] PCI: Using ACPI for IRQ routing1448client # [ 0.491360] NetLabel: unlabeled traffic allowed by default1449worker2 # [ 0.470937] NetLabel: Initializing1450client # [ 0.492182] PCI: Using ACPI for IRQ routing1451worker2 # [ 0.471260] NetLabel: domain hash size = 1281452worker2 # [ 0.472019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1453worker2 # [ 0.472297] NetLabel: unlabeled traffic allowed by default1454worker2 # [ 0.473271] PCI: Using ACPI for IRQ routing1455worker1 # [ 0.525059] pci 0000:00:01.0: vgaarb: setting as boot VGA device1456worker1 # [ 0.525075] pci 0000:00:01.0: vgaarb: bridge control possible1457lb # [ 0.524562] pci 0000:00:01.0: vgaarb: setting as boot VGA device1458worker1 # [ 0.525075] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1459worker1 # [ 0.525086] vgaarb: loaded1460lb # [ 0.525105] pci 0000:00:01.0: vgaarb: bridge control possible1461client # [ 0.535634] pci 0000:00:01.0: vgaarb: setting as boot VGA device1462worker1 # [ 0.525880] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01463lb # [ 0.525105] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1464client # [ 0.536429] pci 0000:00:01.0: vgaarb: bridge control possible1465worker1 # [ 0.526660] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1466lb # [ 0.525113] vgaarb: loaded1467lb # [ 0.525879] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01468client # [ 0.536429] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1469client # [ 0.536438] vgaarb: loaded1470lb # [ 0.526109] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1471client # [ 0.537183] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01472worker2 # [ 0.518126] pci 0000:00:01.0: vgaarb: setting as boot VGA device1473client # [ 0.537435] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1474worker2 # [ 0.518255] pci 0000:00:01.0: vgaarb: bridge control possible1475worker2 # [ 0.518255] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1476worker1 # [ 0.532209] clocksource: Switched to clocksource kvm-clock1477worker2 # [ 0.518264] vgaarb: loaded1478lb # [ 0.530189] clocksource: Switched to clocksource kvm-clock1479worker2 # [ 0.519047] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01480client # [ 0.540515] clocksource: Switched to clocksource kvm-clock1481worker2 # [ 0.519259] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1482worker1 # [ 0.535985] VFS: Disk quotas dquot_6.6.01483lb # [ 0.533565] VFS: Disk quotas dquot_6.6.01484worker1 # [ 0.536705] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1485client # [ 0.543977] VFS: Disk quotas dquot_6.6.01486lb # [ 0.534237] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1487worker1 # [ 0.538022] pnp: PnP ACPI init1488lb # [ 0.535572] pnp: PnP ACPI init1489client # [ 0.544687] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1490worker2 # [ 0.522338] clocksource: Switched to clocksource kvm-clock1491worker1 # [ 0.538843] ACPI: IRQ 4 override to edge(!), high(!)1492client # [ 0.546046] pnp: PnP ACPI init1493lb # [ 0.536290] ACPI: IRQ 4 override to edge(!), high(!)1494worker2 # [ 0.523957] VFS: Disk quotas dquot_6.6.01495worker1 # [ 0.539815] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1496client # [ 0.546860] ACPI: IRQ 4 override to edge(!), high(!)1497lb # [ 0.537228] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1498worker1 # [ 0.541361] pnp: PnP ACPI: found 5 devices1499lb # [ 0.538764] pnp: PnP ACPI: found 5 devices1500client # [ 0.547783] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1501client # [ 0.549348] pnp: PnP ACPI: found 5 devices1502worker2 # [ 0.526429] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1503worker2 # [ 0.527815] pnp: PnP ACPI init1504worker2 # [ 0.528611] ACPI: IRQ 4 override to edge(!), high(!)1505worker2 # [ 0.529620] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1506worker2 # [ 0.531207] pnp: PnP ACPI: found 5 devices1507worker1 # [ 0.549049] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1508lb # [ 0.546349] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1509worker1 # [ 0.550503] clocksource: Switched to clocksource acpi_pm1510lb # [ 0.547769] clocksource: Switched to clocksource acpi_pm1511client # [ 0.557054] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1512worker1 # [ 0.551511] NET: Registered PF_INET protocol family1513lb # [ 0.548691] NET: Registered PF_INET protocol family1514client # [ 0.558472] clocksource: Switched to clocksource acpi_pm1515client # [ 0.559446] NET: Registered PF_INET protocol family1516worker1 # [ 0.552686] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1517lb # [ 0.549804] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1518client # [ 0.560591] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1519worker2 # [ 0.538822] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1520worker2 # [ 0.540333] clocksource: Switched to clocksource acpi_pm1521worker2 # [ 0.541358] NET: Registered PF_INET protocol family1522worker2 # [ 0.542561] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1523lb # [ 0.566303] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1524worker1 # [ 0.569819] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1525client # [ 0.577114] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1526lb # [ 0.567843] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1527worker1 # [ 0.571328] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1528lb # [ 0.569101] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1529client # [ 0.578614] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1530worker1 # [ 0.572611] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1531lb # [ 0.570398] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1532client # [ 0.579930] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1533worker1 # [ 0.573898] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1534lb # [ 0.571591] TCP: Hash tables configured (established 16384 bind 16384)1535client # [ 0.581266] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1536worker1 # [ 0.575095] TCP: Hash tables configured (established 16384 bind 16384)1537client # [ 0.582483] TCP: Hash tables configured (established 16384 bind 16384)1538lb # [ 0.572732] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1539worker2 # [ 0.559384] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1540worker1 # [ 0.576276] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1541lb # [ 0.573994] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1542client # [ 0.583631] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1543worker2 # [ 0.560995] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1544worker1 # [ 0.577483] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1545lb # [ 0.575101] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1546client # [ 0.584919] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1547worker1 # [ 0.578522] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1548worker2 # [ 0.562281] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1549lb # [ 0.576257] NET: Registered PF_UNIX/PF_LOCAL protocol family1550worker1 # [ 0.579682] NET: Registered PF_UNIX/PF_LOCAL protocol family1551client # [ 0.586045] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1552worker2 # [ 0.563576] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1553lb # [ 0.577180] NET: Registered PF_XDP protocol family1554worker1 # [ 0.580647] NET: Registered PF_XDP protocol family1555client # [ 0.587202] NET: Registered PF_UNIX/PF_LOCAL protocol family1556worker2 # [ 0.564739] TCP: Hash tables configured (established 16384 bind 16384)1557lb # [ 0.578070] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1558client # [ 0.588118] NET: Registered PF_XDP protocol family1559worker1 # [ 0.581479] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1560lb # [ 0.579042] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1561worker2 # [ 0.565824] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1562worker1 # [ 0.582438] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1563client # [ 0.588955] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1564lb # [ 0.579994] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1565client # [ 0.589908] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1566worker2 # [ 0.567040] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1567worker1 # [ 0.583397] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1568lb # [ 0.581048] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1569client # [ 0.590881] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1570worker1 # [ 0.584436] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1571worker2 # [ 0.568103] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1572lb # [ 0.582131] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1573worker2 # [ 0.569261] NET: Registered PF_UNIX/PF_LOCAL protocol family1574client # [ 0.591936] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1575worker1 # [ 0.585468] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1576worker2 # [ 0.570168] NET: Registered PF_XDP protocol family1577lb # [ 0.583196] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1578client # [ 0.593024] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1579worker1 # [ 0.586532] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1580worker2 # [ 0.570993] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1581lb # [ 0.584817] ACPI: \_SB_.GSIA: Enabled at IRQ 161582client # [ 0.594077] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1583worker1 # [ 0.588234] ACPI: \_SB_.GSIA: Enabled at IRQ 161584worker2 # [ 0.572007] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1585client # [ 0.595748] ACPI: \_SB_.GSIA: Enabled at IRQ 161586worker2 # [ 0.573005] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1587lb # [ 0.587139] ACPI: \_SB_.GSIB: Enabled at IRQ 171588worker2 # [ 0.574161] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1589worker1 # [ 0.591045] ACPI: \_SB_.GSIB: Enabled at IRQ 171590client # [ 0.598032] ACPI: \_SB_.GSIB: Enabled at IRQ 171591worker2 # [ 0.575298] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1592lb # [ 0.589365] ACPI: \_SB_.GSIC: Enabled at IRQ 181593worker2 # [ 0.576377] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1594worker1 # [ 0.593135] ACPI: \_SB_.GSIC: Enabled at IRQ 181595client # [ 0.600133] ACPI: \_SB_.GSIC: Enabled at IRQ 181596worker2 # [ 0.578116] ACPI: \_SB_.GSIA: Enabled at IRQ 161597lb # [ 0.591305] ACPI: \_SB_.GSID: Enabled at IRQ 191598worker1 # [ 0.595156] ACPI: \_SB_.GSID: Enabled at IRQ 191599client # [ 0.602245] ACPI: \_SB_.GSID: Enabled at IRQ 191600lb # [ 0.593022] PCI: CLS 0 bytes, default 641601worker1 # [ 0.596894] PCI: CLS 0 bytes, default 641602worker2 # [ 0.580864] ACPI: \_SB_.GSIB: Enabled at IRQ 171603client # [ 0.603991] PCI: CLS 0 bytes, default 641604lb # [ 0.593929] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1605client # [ 0.604924] Trying to unpack rootfs image as initramfs...1606worker1 # [ 0.597773] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1607lb # [ 0.595614] Trying to unpack rootfs image as initramfs...1608worker2 # [ 0.582702] ACPI: \_SB_.GSIC: Enabled at IRQ 181609worker1 # [ 0.599468] Trying to unpack rootfs image as initramfs...1610worker2 # [ 0.584670] ACPI: \_SB_.GSID: Enabled at IRQ 191611worker2 # [ 0.586358] PCI: CLS 0 bytes, default 641612client # [ 0.609246] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1613worker2 # [ 0.587336] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1614worker2 # [ 0.589148] Trying to unpack rootfs image as initramfs...1615client # [ 0.650220] Initialise system trusted keyrings1616lb # [ 0.640878] Initialise system trusted keyrings1617worker1 # [ 0.646702] Initialise system trusted keyrings1618lb # [ 0.643539] workingset: timestamp_bits=40 max_order=19 bucket_order=01619client # [ 0.653884] workingset: timestamp_bits=40 max_order=19 bucket_order=01620worker1 # [ 0.647691] workingset: timestamp_bits=40 max_order=19 bucket_order=01621worker2 # [ 0.634496] Initialise system trusted keyrings1622worker2 # [ 0.637135] workingset: timestamp_bits=40 max_order=19 bucket_order=01623client # [ 0.673226] Key type asymmetric registered1624lb # [ 0.664787] Key type asymmetric registered1625lb # [ 0.665526] Asymmetric key parser 'x509' registered1626lb # [ 0.666365] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1627worker1 # [ 0.670948] Key type asymmetric registered1628client # [ 0.677833] Asymmetric key parser 'x509' registered1629worker1 # [ 0.671655] Asymmetric key parser 'x509' registered1630client # [ 0.678698] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1631lb # [ 0.671591] io scheduler mq-deadline registered1632worker1 # [ 0.674306] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1633lb # [ 0.672321] io scheduler kyber registered1634worker2 # [ 0.659151] Key type asymmetric registered1635client # [ 0.681964] io scheduler mq-deadline registered1636client # [ 0.682713] io scheduler kyber registered1637worker2 # [ 0.659877] Asymmetric key parser 'x509' registered1638worker1 # [ 0.677289] io scheduler mq-deadline registered1639worker2 # [ 0.660792] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1640worker1 # [ 0.678020] io scheduler kyber registered1641client # [ 0.685440] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1642lb # [ 0.676065] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1643client # [ 0.686711] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1644lb # [ 0.677336] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1645worker2 # [ 0.666229] io scheduler mq-deadline registered1646worker1 # [ 0.682441] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1647worker2 # [ 0.666996] io scheduler kyber registered1648worker1 # [ 0.683693] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1649lb # [ 0.681219] Linux agpgart interface v0.1031650lb # [ 0.682005] ACPI: bus type drm_connector registered1651client # [ 0.692115] Linux agpgart interface v0.1031652client # [ 0.692898] ACPI: bus type drm_connector registered1653worker2 # [ 0.670640] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1654lb # [ 0.684958] usbcore: registered new interface driver usbserial_generic1655worker2 # [ 0.671901] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1656worker1 # [ 0.689116] Linux agpgart interface v0.1031657lb # [ 0.686077] usbserial: USB Serial support registered for generic1658worker1 # [ 0.689879] ACPI: bus type drm_connector registered1659client # [ 0.696254] usbcore: registered new interface driver usbserial_generic1660client # [ 0.697388] usbserial: USB Serial support registered for generic1661worker2 # [ 0.675844] Linux agpgart interface v0.1031662lb # [ 0.688495] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1663worker2 # [ 0.676600] ACPI: bus type drm_connector registered1664lb # [ 0.688495] Please enable it if your BIOS has the CPPC option.1665worker1 # [ 0.692775] usbcore: registered new interface driver usbserial_generic1666client # [ 0.699840] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1667worker1 # [ 0.693817] usbserial: USB Serial support registered for generic1668lb # [ 0.690706] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1669client # [ 0.699840] Please enable it if your BIOS has the CPPC option.1670client # [ 0.702033] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1671worker2 # [ 0.679549] usbcore: registered new interface driver usbserial_generic1672worker1 # [ 0.696249] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1673worker2 # [ 0.680605] usbserial: USB Serial support registered for generic1674worker1 # [ 0.696249] Please enable it if your BIOS has the CPPC option.1675lb # [ 0.694651] drop_monitor: Initializing network drop monitor service1676worker1 # [ 0.698379] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1677lb # [ 0.695873] NET: Registered PF_INET6 protocol family1678worker2 # [ 0.683090] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1679client # [ 0.706975] drop_monitor: Initializing network drop monitor service1680worker2 # [ 0.683090] Please enable it if your BIOS has the CPPC option.1681client # [ 0.708185] NET: Registered PF_INET6 protocol family1682worker2 # [ 0.685402] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1683worker1 # [ 0.702495] drop_monitor: Initializing network drop monitor service1684lb # [ 0.700578] Segment Routing with IPv61685worker1 # [ 0.703690] NET: Registered PF_INET6 protocol family1686lb # [ 0.701211] In-situ OAM (IOAM) with IPv61687client # [ 0.711298] Segment Routing with IPv61688lb # [ 0.702313] IPI shorthand broadcast: enabled1689worker2 # [ 0.690228] drop_monitor: Initializing network drop monitor service1690client # [ 0.713840] In-situ OAM (IOAM) with IPv61691worker2 # [ 0.691451] NET: Registered PF_INET6 protocol family1692worker1 # [ 0.708339] Segment Routing with IPv61693client # [ 0.714975] IPI shorthand broadcast: enabled1694worker1 # [ 0.709142] In-situ OAM (IOAM) with IPv61695worker2 # [ 0.694544] Segment Routing with IPv61696worker1 # [ 0.712600] IPI shorthand broadcast: enabled1697worker2 # [ 0.697112] In-situ OAM (IOAM) with IPv61698lb # [ 0.710604] sched_clock: Marking stable (576013342, 134462888)->(788823085, -78346855)1699worker2 # [ 0.698219] IPI shorthand broadcast: enabled1700client # [ 0.723203] sched_clock: Marking stable (578015478, 144792136)->(800938505, -78130891)1701lb # [ 0.714698] registered taskstats version 11702lb # [ 0.715661] Loading compiled-in X.509 certificates1703worker1 # [ 0.718917] sched_clock: Marking stable (583015499, 135209864)->(793360083, -75134720)1704client # [ 0.727084] registered taskstats version 11705client # [ 0.728104] Loading compiled-in X.509 certificates1706worker2 # [ 0.706337] sched_clock: Marking stable (573014409, 133056275)->(781536342, -75465658)1707worker1 # [ 0.725333] registered taskstats version 11708worker1 # [ 0.726292] Loading compiled-in X.509 certificates1709worker2 # [ 0.711348] registered taskstats version 11710worker2 # [ 0.712359] Loading compiled-in X.509 certificates1711lb # [ 0.734490] Demotion targets for Node 0: null1712lb # [ 0.735412] Key type .fscrypt registered1713client # [ 0.746439] Demotion targets for Node 0: null1714lb # [ 0.738481] Key type fscrypt-provisioning registered1715lb # [ 0.739489] ima: No TPM chip found, activating TPM-bypass!1716client # [ 0.749851] Key type .fscrypt registered1717worker1 # [ 0.743252] Demotion targets for Node 0: null1718lb # [ 0.740437] ima: Allocated hash algorithm: sha11719client # [ 0.750517] Key type fscrypt-provisioning registered1720worker1 # [ 0.744180] Key type .fscrypt registered1721client # [ 0.751467] ima: No TPM chip found, activating TPM-bypass!1722lb # [ 0.742514] ima: No architecture policies found1723worker2 # [ 0.730085] Demotion targets for Node 0: null1724lb # [ 0.743487] PM: Magic number: 10:890:1631725worker2 # [ 0.730981] Key type .fscrypt registered1726client # [ 0.753819] ima: Allocated hash algorithm: sha11727worker1 # [ 0.747234] Key type fscrypt-provisioning registered1728client # [ 0.754598] ima: No architecture policies found1729lb # [ 0.745083] RAS: Correctable Errors collector initialized.1730worker1 # [ 0.748199] ima: No TPM chip found, activating TPM-bypass!1731worker1 # [ 0.749105] ima: Allocated hash algorithm: sha11732client # [ 0.756825] PM: Magic number: 10:890:1631733worker2 # [ 0.734075] Key type fscrypt-provisioning registered1734worker1 # [ 0.751270] ima: No architecture policies found1735worker2 # [ 0.735033] ima: No TPM chip found, activating TPM-bypass!1736client # [ 0.758441] RAS: Correctable Errors collector initialized.1737worker1 # [ 0.752270] PM: Magic number: 10:890:1631738worker2 # [ 0.735993] ima: Allocated hash algorithm: sha11739worker1 # [ 0.754146] RAS: Correctable Errors collector initialized.1740worker2 # [ 0.738108] ima: No architecture policies found1741worker2 # [ 0.739032] PM: Magic number: 10:890:1631742lb # [ 0.754117] clk: Disabling unused clocks1743worker2 # [ 0.742743] RAS: Correctable Errors collector initialized.1744lb # [ 0.757494] PM: genpd: Disabling unused power domains1745client # [ 0.767588] clk: Disabling unused clocks1746client # [ 0.770849] PM: genpd: Disabling unused power domains1747worker1 # [ 0.765053] clk: Disabling unused clocks1748worker1 # [ 0.767255] PM: genpd: Disabling unused power domains1749worker2 # [ 0.751411] clk: Disabling unused clocks1750worker2 # [ 0.754095] PM: genpd: Disabling unused power domains1751client # [ 0.932559] Freeing initrd memory: 29056K1752lb # [ 0.923620] Freeing initrd memory: 29064K1753client # [ 0.935705] Freeing unused decrypted memory: 2028K1754lb # [ 0.926830] Freeing unused decrypted memory: 2028K1755worker2 # [ 0.913715] Freeing initrd memory: 29060K1756client # [ 0.938327] Freeing unused kernel image (initmem) memory: 3652K1757lb # [ 0.929418] Freeing unused kernel image (initmem) memory: 3652K1758worker2 # [ 0.916863] Freeing unused decrypted memory: 2028K1759client # [ 0.939619] Write protecting the kernel read-only data: 32768k1760lb # [ 0.930534] Write protecting the kernel read-only data: 32768k1761client # [ 0.941453] Freeing unused kernel image (text/rodata gap) memory: 1184K1762lb # [ 0.932353] Freeing unused kernel image (text/rodata gap) memory: 1184K1763worker2 # [ 0.919412] Freeing unused kernel image (initmem) memory: 3652K1764client # [ 0.942986] Freeing unused kernel image (rodata/data gap) memory: 720K1765worker2 # [ 0.920518] Write protecting the kernel read-only data: 32768k1766worker1 # [ 0.936922] Freeing initrd memory: 29056K1767lb # [ 0.933903] Freeing unused kernel image (rodata/data gap) memory: 720K1768worker2 # [ 0.922317] Freeing unused kernel image (text/rodata gap) memory: 1184K1769worker2 # [ 0.923841] Freeing unused kernel image (rodata/data gap) memory: 720K1770worker1 # [ 0.940458] Freeing unused decrypted memory: 2028K1771worker1 # [ 0.943462] Freeing unused kernel image (initmem) memory: 3652K1772worker1 # [ 0.944579] Write protecting the kernel read-only data: 32768k1773worker1 # [ 0.946500] Freeing unused kernel image (text/rodata gap) memory: 1184K1774worker1 # [ 0.947993] Freeing unused kernel image (rodata/data gap) memory: 720K1775client # [ 0.993728] x86/mm: Checked W+X mappings: passed, no W+X pages found.1776lb # [ 0.984708] x86/mm: Checked W+X mappings: passed, no W+X pages found.1777client # [ 0.995097] Run /init as init process1778lb # [ 0.985753] Run /init as init process1779worker2 # [ 0.974792] x86/mm: Checked W+X mappings: passed, no W+X pages found.1780worker2 # [ 0.975915] Run /init as init process1781client # [ 1.005614] systemd[1]: Inserted module 'autofs4'1782worker1 # [ 0.998877] x86/mm: Checked W+X mappings: passed, no W+X pages found.1783lb # [ 0.996254] systemd[1]: Inserted module 'autofs4'1784worker1 # [ 0.999937] Run /init as init process1785worker2 # [ 0.986742] systemd[1]: Inserted module 'autofs4'1786worker1 # [ 1.011652] systemd[1]: Inserted module 'autofs4'1787lb # [ 1.015903] fuse: init (API version 7.45)1788client # [ 1.025666] fuse: init (API version 7.45)1789worker2 # [ 1.006562] fuse: init (API version 7.45)1790lb # [ 1.023226] ACPI: \_SB_.GSIG: Enabled at IRQ 221791client # [ 1.033131] ACPI: \_SB_.GSIG: Enabled at IRQ 221792lb # [ 1.025697] ACPI: \_SB_.GSIH: Enabled at IRQ 231793client # [ 1.035756] ACPI: \_SB_.GSIH: Enabled at IRQ 231794worker2 # [ 1.013904] ACPI: \_SB_.GSIG: Enabled at IRQ 221795lb # [ 1.028707] ACPI: \_SB_.GSIE: Enabled at IRQ 201796client # [ 1.038776] ACPI: \_SB_.GSIE: Enabled at IRQ 201797worker1 # [ 1.032451] fuse: init (API version 7.45)1798worker2 # [ 1.016526] ACPI: \_SB_.GSIH: Enabled at IRQ 231799lb # [ 1.030686] ACPI: \_SB_.GSIF: Enabled at IRQ 211800client # [ 1.040735] ACPI: \_SB_.GSIF: Enabled at IRQ 211801worker2 # [ 1.020033] ACPI: \_SB_.GSIE: Enabled at IRQ 201802worker2 # [ 1.022314] ACPI: \_SB_.GSIF: Enabled at IRQ 211803lb # [ 1.035533] virtiofs virtio5: discovered new tag: nix-store1804client # [ 1.045834] virtiofs virtio5: discovered new tag: nix-store1805lb # [ 1.037058] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1806worker1 # [ 1.040807] ACPI: \_SB_.GSIG: Enabled at IRQ 221807client # [ 1.047398] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1808worker1 # [ 1.043288] ACPI: \_SB_.GSIH: Enabled at IRQ 231809worker2 # [ 1.027393] virtiofs virtio5: discovered new tag: nix-store1810worker2 # [ 1.029099] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1811worker1 # [ 1.046683] ACPI: \_SB_.GSIE: Enabled at IRQ 201812lb # [ 1.043699] virtiofs virtio6: discovered new tag: shared1813client # [ 1.053953] virtiofs virtio6: discovered new tag: shared1814lb # [ 1.045139] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1815worker1 # [ 1.048976] ACPI: \_SB_.GSIF: Enabled at IRQ 211816client # [ 1.055430] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1817lb # [ 1.048538] virtiofs virtio7: discovered new tag: xchg1818client # [ 1.058771] virtiofs virtio7: discovered new tag: xchg1819worker2 # [ 1.036536] virtiofs virtio6: discovered new tag: shared1820lb # [ 1.050035] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1821client # [ 1.060213] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1822worker2 # [ 1.038084] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1823worker1 # [ 1.054875] virtiofs virtio5: discovered new tag: nix-store1824worker1 # [ 1.056506] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1825worker2 # [ 1.041309] virtiofs virtio7: discovered new tag: xchg1826worker2 # [ 1.042820] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1827worker1 # [ 1.064002] virtiofs virtio6: discovered new tag: shared1828worker1 # [ 1.065558] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1829worker1 # [ 1.070136] virtiofs virtio7: discovered new tag: xchg1830worker1 # [ 1.071782] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1831lb # [ 1.072286] systemd[1]: Successfully made /usr/ read-only.1832client # [ 1.082648] systemd[1]: Successfully made /usr/ read-only.1833worker2 # [ 1.064199] systemd[1]: Successfully made /usr/ read-only.1834worker1 # [ 1.093150] systemd[1]: Successfully made /usr/ read-only.1835client # [ 1.418216] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1836client # [ 1.423341] systemd[1]: Detected virtualization kvm.1837client # [ 1.424211] systemd[1]: Detected architecture x86-64.1838client # [ 1.425106] systemd[1]: Running in initrd.1839client # [ 1.426134] systemd[1]: Initializing machine ID from random generator.1840client # [ 1.427318] systemd[1]: Hostname set to <client>.1841lb # [ 1.409002] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1842lb # [ 1.421139] systemd[1]: Detected virtualization kvm.1843lb # [ 1.423347] systemd[1]: Detected architecture x86-64.1844worker2 # [ 1.400837] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1845lb # [ 1.425500] systemd[1]: Running in initrd.1846worker2 # [ 1.412975] systemd[1]: Detected virtualization kvm.1847worker2 # [ 1.415168] systemd[1]: Detected architecture x86-64.1848lb # [ 1.427928] systemd[1]: Initializing machine ID from random generator.1849worker2 # [ 1.417383] systemd[1]: Running in initrd.1850worker1 # [ 1.428670] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1851lb # [ 1.430825] systemd[1]: Hostname set to <lb>.1852worker1 # [ 1.435775] systemd[1]: Detected virtualization kvm.1853worker2 # [ 1.419867] systemd[1]: Initializing machine ID from random generator.1854worker1 # [ 1.438003] systemd[1]: Detected architecture x86-64.1855worker2 # [ 1.422814] systemd[1]: Hostname set to <worker2>.1856worker1 # [ 1.440175] systemd[1]: Running in initrd.1857worker1 # [ 1.442675] systemd[1]: Initializing machine ID from random generator.1858worker1 # [ 1.445550] systemd[1]: Hostname set to <worker1>.1859client # [ 1.628845] systemd[1]: bpf-restrict-fs: LSM BPF program attached1860lb # [ 1.652617] systemd[1]: bpf-restrict-fs: LSM BPF program attached1861worker2 # [ 1.646445] systemd[1]: bpf-restrict-fs: LSM BPF program attached1862worker1 # [ 1.665477] systemd[1]: bpf-restrict-fs: LSM BPF program attached1863client # [ 1.681708] systemd[1]: Queued start job for default target Initrd Default Target.1864client # [ 1.686150] systemd[1]: Created slice Slice /system/modprobe.1865client # [ 1.687276] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1866client # [ 1.688680] systemd[1]: Expecting device /dev/disk/by-label/nixos...1867client # [ 1.689741] systemd[1]: Reached target Path Units.1868client # [ 1.690584] systemd[1]: Reached target Slice Units.1869client # [ 1.691427] systemd[1]: Reached target Swaps.1870client # [ 1.692188] systemd[1]: Reached target Timer Units.1871client # [ 1.693107] systemd[1]: Listening on D-Bus System Message Bus Socket.1872client # [ 1.694242] systemd[1]: Listening on Journal Socket (/dev/log).1873client # [ 1.695330] systemd[1]: Listening on Journal Sockets.1874client # [ 1.696274] systemd[1]: Listening on udev Control Socket.1875client # [ 1.697277] systemd[1]: Listening on udev Kernel Socket.1876client # [ 1.698184] systemd[1]: Reached target Socket Units.1877client # [ 1.699869] systemd[1]: Starting Create List of Static Device Nodes...1878client # [ 1.703641] systemd[1]: Starting Load Kernel Module configfs...1879lb # [ 1.704695] systemd[1]: Queued start job for default target Initrd Default Target.1880lb # [ 1.708767] systemd[1]: Created slice Slice /system/modprobe.1881lb # [ 1.709947] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1882worker2 # [ 1.697736] systemd[1]: Queued start job for default target Initrd Default Target.1883lb # [ 1.711398] systemd[1]: Expecting device /dev/disk/by-label/nixos...1884lb # [ 1.712516] systemd[1]: Reached target Path Units.1885lb # [ 1.713372] systemd[1]: Reached target Slice Units.1886lb # [ 1.714343] systemd[1]: Reached target Swaps.1887lb # [ 1.715230] systemd[1]: Reached target Timer Units.1888client # [ 1.713898] systemd[1]: Starting Journal Service...1889lb # [ 1.716238] systemd[1]: Listening on D-Bus System Message Bus Socket.1890worker2 # [ 1.702836] systemd[1]: Created slice Slice /system/modprobe.1891worker1 # [ 1.719607] systemd[1]: Queued start job for default target Initrd Default Target.1892lb # [ 1.717519] systemd[1]: Listening on Journal Socket (/dev/log).1893worker2 # [ 1.704607] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1894lb # [ 1.718647] systemd[1]: Listening on Journal Sockets.1895lb # [ 1.719667] systemd[1]: Listening on udev Control Socket.1896worker2 # [ 1.706416] systemd[1]: Expecting device /dev/disk/by-label/nixos...1897worker1 # [ 1.723507] systemd[1]: Created slice Slice /system/modprobe.1898lb # [ 1.720705] systemd[1]: Listening on udev Kernel Socket.1899worker2 # [ 1.707796] systemd[1]: Reached target Path Units.1900lb # [ 1.721657] systemd[1]: Reached target Socket Units.1901worker1 # [ 1.724695] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1902worker2 # [ 1.708915] systemd[1]: Reached target Slice Units.1903worker1 # [ 1.726077] systemd[1]: Expecting device /dev/disk/by-label/nixos...1904worker2 # [ 1.710043] systemd[1]: Reached target Swaps.1905lb # [ 1.723358] systemd[1]: Starting Create List of Static Device Nodes...1906worker1 # [ 1.727193] systemd[1]: Reached target Path Units.1907worker2 # [ 1.711107] systemd[1]: Reached target Timer Units.1908worker1 # [ 1.728105] systemd[1]: Reached target Slice Units.1909worker1 # [ 1.729019] systemd[1]: Reached target Swaps.1910worker2 # [ 1.712345] systemd[1]: Listening on D-Bus System Message Bus Socket.1911worker1 # [ 1.729870] systemd[1]: Reached target Timer Units.1912lb # [ 1.727279] systemd[1]: Starting Load Kernel Module configfs...1913worker2 # [ 1.713905] systemd[1]: Listening on Journal Socket (/dev/log).1914client # [ 1.737360] systemd[1]: Starting Load Kernel Modules...1915worker1 # [ 1.730840] systemd[1]: Listening on D-Bus System Message Bus Socket.1916worker2 # [ 1.715435] systemd[1]: Listening on Journal Sockets.1917worker1 # [ 1.731995] systemd[1]: Listening on Journal Socket (/dev/log).1918worker1 # [ 1.733193] systemd[1]: Listening on Journal Sockets.1919worker2 # [ 1.716745] systemd[1]: Listening on udev Control Socket.1920worker1 # [ 1.734197] systemd[1]: Listening on udev Control Socket.1921worker2 # [ 1.718078] systemd[1]: Listening on udev Kernel Socket.1922worker1 # [ 1.735264] systemd[1]: Listening on udev Kernel Socket.1923worker2 # [ 1.719310] systemd[1]: Reached target Socket Units.1924client # [ 1.741930] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1925worker1 # [ 1.736198] systemd[1]: Reached target Socket Units.1926worker1 # [ 1.737986] systemd[1]: Starting Create List of Static Device Nodes...1927worker2 # [ 1.721705] systemd[1]: Starting Create List of Static Device Nodes...1928worker1 # [ 1.742294] systemd[1]: Starting Load Kernel Module configfs...1929client # [ 1.750052] systemd[1]: Starting Coldplug All udev Devices...1930worker2 # [ 1.728257] systemd[1]: Starting Load Kernel Module configfs...1931lb # [ 1.734155] systemd[1]: Starting Journal Service...1932client # [ 1.763319] systemd-journald[66]: Collecting audit messages is disabled.1933client # [ 1.764936] systemd[1]: Finished Create List of Static Device Nodes.1934client # [ 1.767426] systemd[1]: modprobe@configfs.service: Deactivated successfully.1935worker1 # [ 1.751865] systemd[1]: Starting Journal Service...1936lb # [ 1.761654] systemd[1]: Starting Load Kernel Modules...1937worker2 # [ 1.737268] systemd[1]: Starting Journal Service...1938client # [ 1.776204] systemd[1]: Finished Load Kernel Module configfs.1939lb # [ 1.767591] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1940client # [ 1.781238] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1941worker1 # [ 1.777422] systemd[1]: Starting Load Kernel Modules...1942lb # [ 1.774658] systemd[1]: Starting Coldplug All udev Devices...1943worker1 # [ 1.781325] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1944worker2 # [ 1.765769] systemd[1]: Starting Load Kernel Modules...1945client # [ 1.791869] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1946lb # [ 1.783283] systemd-journald[66]: Collecting audit messages is disabled.1947worker1 # [ 1.788014] systemd[1]: Starting Coldplug All udev Devices...1948lb # [ 1.786895] systemd[1]: Finished Create List of Static Device Nodes.1949worker2 # [ 1.773214] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1950client # [ 1.801287] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1951lb # [ 1.793955] systemd[1]: modprobe@configfs.service: Deactivated successfully.1952worker2 # [ 1.781287] systemd[1]: Starting Coldplug All udev Devices...1953worker1 # [ 1.800794] systemd-journald[66]: Collecting audit messages is disabled.1954worker1 # [ 1.804293] systemd[1]: Finished Create List of Static Device Nodes.1955client # [ 1.810986] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1956lb # [ 1.801874] systemd[1]: Finished Load Kernel Module configfs.1957worker2 # [ 1.790345] systemd-journald[66]: Collecting audit messages is disabled.1958worker1 # [ 1.806858] systemd[1]: modprobe@configfs.service: Deactivated successfully.1959lb # [ 1.806897] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1960worker2 # [ 1.797465] systemd[1]: Finished Create List of Static Device Nodes.1961worker1 # [ 1.815766] systemd[1]: Finished Load Kernel Module configfs.1962worker2 # [ 1.802732] systemd[1]: modprobe@configfs.service: Deactivated successfully.1963worker1 # [ 1.821651] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1964lb # [ 1.820547] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1965client # [ 1.831330] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1966worker2 # [ 1.811512] systemd[1]: Finished Load Kernel Module configfs.1967worker2 # [ 1.813540] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1968lb # [ 1.828801] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1969client # [ 1.839997] systemd[1]: Starting Create Static Device Nodes in /dev...1970worker1 # [ 1.835301] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1971lb # [ 1.837544] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1972worker1 # [ 1.841695] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1973worker2 # [ 1.826097] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1974client # [ 1.852337] systemd[1]: Finished Load Kernel Modules.1975worker2 # [ 1.831893] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1976worker1 # [ 1.853600] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1977client # [ 1.861024] systemd[1]: Starting Apply Kernel Variables...1978worker2 # [ 1.844256] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1979lb # [ 1.861007] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1980lb # [ 1.869661] systemd[1]: Starting Create Static Device Nodes in /dev...1981client # [ 1.880106] systemd[1]: Finished Create Static Device Nodes in /dev.1982worker1 # [ 1.875796] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1983client # [ 1.739575] systemd-modules-load[67]: Inserted module 'dm_mod'[ 1.885632] systemd[1]: Started Journal Service.1984client # 1985client # [ 1.744313] systemd-modules-load[67]: Inserted module 'virtio_balloon'1986client # [ 1.745372] systemd-modules-load[67]: Inserted module 'virtio_gpu'1987lb # [ 1.880999] systemd[1]: Finished Load Kernel Modules.1988worker1 # [ 1.884423] systemd[1]: Starting Create Static Device Nodes in /dev...1989worker2 # [ 1.872150] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1990client # [ 1.751255] systemd[1]: Finished Apply Kernel Variables.1991lb # [ 1.888851] systemd[1]: Starting Apply Kernel Variables...1992client # [ 1.756920] systemd[1]: Reached target Preparation for Local File Systems.1993client # [ 1.758049] systemd[1]: Reached target Local File Systems.1994worker1 # [ 1.896794] systemd[1]: Finished Load Kernel Modules.1995worker2 # [ 1.882286] systemd[1]: Starting Create Static Device Nodes in /dev...1996client # [ 1.760959] systemd[1]: Starting Create System Files and Directories...1997worker1 # [ 1.905297] systemd[1]: Starting Apply Kernel Variables...1998client # [ 1.768075] systemd[1]: Starting Rule-based Manager for Device Events and Files...1999worker2 # [ 1.892618] systemd[1]: Finished Load Kernel Modules.2000lb # [ 1.906606] systemd[1]: Started Journal Service.
loading 2,913 hidden lines…
4914worker2 # [ 45.568055] nix-grpc-daemon[880]: time=2026-09-23T05:12:35.732Z level=INFO msg="Uploading qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a (128B)"4915lb # [ 45.603133] niks3-server[977]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst4916lb # [ 45.608115] niks3-server[977]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=log/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv4917lb # [ 45.617707] niks3-server[977]: 2026/09/23 05:12:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4918lb # [ 45.623591] niks3-server[977]: 2026/09/23 05:12:35 INFO Signed narinfos id=10 count=14919worker2 # [ 45.614988] nix-grpc-daemon[880]: time=2026-09-23T05:12:35.780Z level=INFO msg="Uploading 1 narinfos"4920lb # [ 45.629963] niks3-server[977]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=qi07z82gylcaha6pl0vfp8s3j6f2gl91.ls4921lb # [ 45.636199] niks3-server[977]: 2026/09/23 05:12:35 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4922lb # [ 45.644108] niks3-server[977]: 2026/09/23 05:12:35 INFO Completed upload id=104923worker2 # [ 45.632373] nix-grpc-daemon[880]: time=2026-09-23T05:12:35.797Z level=INFO msg="Upload complete. (203ms)"4924lb # [ 45.645948] niks3-server[977]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=qi07z82gylcaha6pl0vfp8s3j6f2gl91.narinfo4925worker2 # [ 45.637803] nix-grpc-daemon[783]: ts=2026-09-23T05:12:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=1 drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv assign_id=5 outputs=14926worker1 # [ 45.659512] nix-grpc-daemon[780]: ts=2026-09-23T05:12:35Z level=debug event=want client=4 drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv cached=04927worker1 # [ 45.663717] nix-grpc-daemon[780]: ts=2026-09-23T05:12:35Z level=debug event=assigned drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv worker=192.168.1.4:50051 clients=14928worker2 # [ 45.652108] nix-daemon[851]: reaped child process 898, status = succeeded4929worker2 # [ 45.689382] nix-grpc-daemon[783]: ts=2026-09-23T05:12:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398044930worker2 # [ 45.702270] nix-grpc-daemon[783]: ts=2026-09-23T05:12:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398044931worker2 # [ 45.715565] nix-grpc-daemon[880]: time=2026-09-23T05:12:35.880Z level=INFO msg="All 1 paths already cached"4932client # copying 1 paths to 'lb:50051'...4933client # copying path '/nix/store/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv' to 'lb:50051'...4934worker2 # [ 45.723804] nix-grpc-daemon[783]: ts=2026-09-23T05:12:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398044935worker2 # [ 45.731679] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4936worker2 # [ 45.830974] nix-daemon[851]: accepted connection from pid 923, user nix-grpc-daemon (trusted)4937worker2 # [ 45.846355] nix-daemon[851]: reaped child process 926, status = succeeded4938lb # [ 45.863245] niks3-server[977]: 2026/09/23 05:12:35 INFO Received uploads request method=POST path=/api/pending_closures4939worker2 # [ 45.858374] nix-grpc-daemon[880]: time=2026-09-23T05:12:36.023Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4940worker2 # [ 45.860024] nix-grpc-daemon[880]: time=2026-09-23T05:12:36.025Z level=INFO msg="Uploading 08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv (624B)"4941lb # [ 45.893251] niks3-server[977]: 2026/09/23 05:12:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4942lb # [ 45.897066] niks3-server[977]: 2026/09/23 05:12:35 INFO Signed narinfos id=11 count=14943worker2 # [ 45.888193] nix-grpc-daemon[880]: time=2026-09-23T05:12:36.053Z level=INFO msg="Uploading 1 narinfos"4944lb # [ 45.900529] niks3-server[977]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=08wvq73c5i2kpl6g578glyb1rcqhv2iw.ls4945lb # [ 45.906894] niks3-server[977]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=nar/1m1mvw8bmb91glp26qj699lxqv98q2iz8wm4ik3miz0xj6vi3mw6.nar.zst4946lb # [ 45.917719] niks3-server[977]: 2026/09/23 05:12:35 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4947worker2 # [ 45.912572] nix-grpc-daemon[880]: time=2026-09-23T05:12:36.077Z level=INFO msg="Upload complete. (167ms)"4948lb # [ 45.924476] niks3-server[977]: 2026/09/23 05:12:35 INFO Completed upload id=114949worker2 # [ 45.914820] nix-grpc-daemon[783]: ts=2026-09-23T05:12:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=0 paths=1 nar_bytes_in=6244950lb # [ 45.927357] niks3-server[977]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=08wvq73c5i2kpl6g578glyb1rcqhv2iw.narinfo4951worker2 # [ 45.918460] nix-grpc-daemon[783]: ts=2026-09-23T05:12:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398044952worker2 # [ 45.927354] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4953worker2 # [ 45.931094] nix-daemon[851]: reaped child process 921, status = succeeded4954worker2 # [ 45.934807] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4955client # building '/nix/store/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv' on lb:50051...4956client # farm-b> worker2: building 08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv4957client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4958client # farm-b> LOG-farm-b-sig4959worker2 # [ 47.171337] nix-daemon[851]: reaped child process 930, status = succeeded4960worker2 # [ 47.250722] nix-daemon[851]: accepted connection from pid 946, user nix-grpc-daemon (trusted)4961worker2 # [ 47.265095] nix-daemon[851]: reaped child process 949, status = succeeded4962lb # [ 47.282805] niks3-server[977]: 2026/09/23 05:12:37 INFO Received uploads request method=POST path=/api/pending_closures4963worker2 # [ 47.280519] nix-grpc-daemon[880]: time=2026-09-23T05:12:37.445Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4964worker2 # [ 47.282831] nix-grpc-daemon[880]: time=2026-09-23T05:12:37.447Z level=INFO msg="Uploading 2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b (176B)"4965lb # [ 47.330695] niks3-server[977]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=nar/0y3xlf15z3z540fgrc9sgnn2rl0aj3w6x9mjqd50hhqgmaxbiivd.nar.zst4966lb # [ 47.336430] niks3-server[977]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.ls4967lb # [ 47.344357] niks3-server[977]: 2026/09/23 05:12:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4968lb # [ 47.348415] niks3-server[977]: 2026/09/23 05:12:37 INFO Signed narinfos id=12 count=14969worker2 # [ 47.341512] nix-grpc-daemon[880]: time=2026-09-23T05:12:37.506Z level=INFO msg="Uploading 1 narinfos"4970lb # [ 47.362477] niks3-server[977]: 2026/09/23 05:12:37 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4971lb # [ 47.392819] niks3-server[977]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=log/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv4972lb # [ 47.399832] niks3-server[977]: 2026/09/23 05:12:37 INFO Completed upload id=124973worker2 # [ 47.387692] nix-grpc-daemon[880]: time=2026-09-23T05:12:37.552Z level=INFO msg="Upload complete. (245ms)"4974lb # [ 47.404064] niks3-server[977]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.narinfo4975worker2 # [ 47.394461] nix-grpc-daemon[783]: ts=2026-09-23T05:12:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=1 drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv assign_id=6 outputs=14976worker1 # [ 47.417530] nix-grpc-daemon[780]: ts=2026-09-23T05:12:37Z level=debug event=want client=4 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv cached=04977worker1 # [ 47.421658] nix-grpc-daemon[780]: ts=2026-09-23T05:12:37Z level=debug event=assigned drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv worker=192.168.1.4:50051 clients=14978worker2 # [ 47.410384] nix-daemon[851]: reaped child process 928, status = succeeded4979worker2 # [ 47.448163] nix-grpc-daemon[783]: ts=2026-09-23T05:12:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398044980worker2 # [ 47.461657] nix-grpc-daemon[783]: ts=2026-09-23T05:12:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398044981worker2 # [ 47.475419] nix-grpc-daemon[880]: time=2026-09-23T05:12:37.640Z level=INFO msg="All 2 paths already cached"4982client # copying 1 paths to 'lb:50051'...4983client # copying path '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' to 'lb:50051'...4984worker2 # [ 47.484403] nix-grpc-daemon[783]: ts=2026-09-23T05:12:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398044985worker2 # [ 47.492428] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)4986worker2 # [ 47.593448] nix-daemon[851]: accepted connection from pid 953, user nix-grpc-daemon (trusted)4987worker2 # [ 47.606568] nix-daemon[851]: reaped child process 956, status = succeeded4988lb # [ 47.624681] niks3-server[977]: 2026/09/23 05:12:37 INFO Received uploads request method=POST path=/api/pending_closures4989worker2 # [ 47.622366] nix-grpc-daemon[880]: time=2026-09-23T05:12:37.787Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4990worker2 # [ 47.623833] nix-grpc-daemon[880]: time=2026-09-23T05:12:37.789Z level=INFO msg="Uploading cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv (752B)"4991lb # [ 47.660889] niks3-server[977]: 2026/09/23 05:12:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4992lb # [ 47.670112] niks3-server[977]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=nar/0w21abkvlzg18nqkl2x4zxs7d0376kbld1n0fkr68vcpqpq1g5zg.nar.zst4993lb # [ 47.673527] niks3-server[977]: 2026/09/23 05:12:37 INFO Signed narinfos id=13 count=14994worker2 # [ 47.664276] nix-grpc-daemon[880]: time=2026-09-23T05:12:37.829Z level=INFO msg="Uploading 1 narinfos"4995lb # [ 47.680471] niks3-server[977]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.ls4996lb # [ 47.688972] niks3-server[977]: 2026/09/23 05:12:37 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4997lb # [ 47.692184] niks3-server[977]: 2026/09/23 05:12:37 INFO Completed upload id=134998worker2 # [ 47.680678] nix-grpc-daemon[880]: time=2026-09-23T05:12:37.846Z level=INFO msg="Upload complete. (174ms)"4999lb # [ 47.694298] niks3-server[977]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.narinfo5000worker2 # [ 47.684548] nix-grpc-daemon[783]: ts=2026-09-23T05:12:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=0 paths=1 nar_bytes_in=7525001worker2 # [ 47.689271] nix-grpc-daemon[783]: ts=2026-09-23T05:12:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398045002worker2 # [ 47.700323] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5003worker2 # [ 47.704259] nix-daemon[851]: reaped child process 951, status = succeeded5004worker2 # [ 47.708612] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5005client # building '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' on lb:50051...5006client # farm-top> worker2: building cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv5007client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5008client # farm-top> LOG-farm-top-sig5009worker2 # [ 49.143218] nix-daemon[851]: reaped child process 960, status = succeeded5010worker2 # [ 49.227631] nix-daemon[851]: accepted connection from pid 976, user nix-grpc-daemon (trusted)5011worker2 # [ 49.240548] nix-daemon[851]: reaped child process 979, status = succeeded5012lb # [ 49.259960] niks3-server[977]: 2026/09/23 05:12:39 INFO Received uploads request method=POST path=/api/pending_closures5013worker2 # [ 49.257463] nix-grpc-daemon[880]: time=2026-09-23T05:12:39.422Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5014worker2 # [ 49.259539] nix-grpc-daemon[880]: time=2026-09-23T05:12:39.424Z level=INFO msg="Uploading dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top (232B)"5015lb # [ 49.306459] niks3-server[977]: 2026/09/23 05:12:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign5016lb # [ 49.314872] niks3-server[977]: 2026/09/23 05:12:39 INFO Signed narinfos id=14 count=15017worker2 # [ 49.305807] nix-grpc-daemon[880]: time=2026-09-23T05:12:39.470Z level=INFO msg="Uploading 1 narinfos"5018lb # [ 49.323827] niks3-server[977]: 2026/09/23 05:12:39 INFO Registered completed upload object_key=log/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv5019lb # [ 49.328279] niks3-server[977]: 2026/09/23 05:12:39 INFO Registered completed upload object_key=nar/0h4zvhqjxsvdqli2l6i5ajh5sh40hz57gzchdirnvbisq2aks0k8.nar.zst5020lb # [ 49.332892] niks3-server[977]: 2026/09/23 05:12:39 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.ls5021lb # [ 49.341667] niks3-server[977]: 2026/09/23 05:12:39 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete5022lb # [ 49.345280] niks3-server[977]: 2026/09/23 05:12:39 INFO Completed upload id=145023lb # [ 49.348160] niks3-server[977]: 2026/09/23 05:12:39 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.narinfo5024worker2 # [ 49.335320] nix-grpc-daemon[880]: time=2026-09-23T05:12:39.500Z level=INFO msg="Upload complete. (217ms)"5025worker2 # [ 49.341598] nix-grpc-daemon[783]: ts=2026-09-23T05:12:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=1 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv assign_id=7 outputs=15026worker2 # [ 49.355450] nix-daemon[851]: reaped child process 958, status = succeeded5027client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig >&2, in 5.29 seconds)5028client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig outPath5029client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig outPath, in 0.12 seconds)5030client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5031worker1 # [ 49.606757] nix-grpc-daemon[780]: ts=2026-09-23T05:12:39Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:487325032client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked):5033client # /nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a5034client # /nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b5035client # /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5036worker1 # [ 49.648759] nix-grpc-daemon[780]: ts=2026-09-23T05:12:39Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:487325037worker1 # [ 49.655400] nix-grpc-daemon[780]: copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'http://lb:5751'...5038worker1 # [ 49.670329] nix-grpc-daemon[780]: copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'http://lb:5751'...5039worker1 # [ 49.682475] nix-grpc-daemon[780]: copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'http://lb:5751'...5040client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.21 seconds)5041client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5042worker2 # [ 49.817618] nix-grpc-daemon[783]: ts=2026-09-23T05:12:39Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:370365043worker2 # [ 49.823268] nix-grpc-daemon[783]: ts=2026-09-23T05:12:39Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:370365044client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.18 seconds)5045client: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5046client # finding garbage collector roots...5047client # removing stale temporary roots file "/nix/var/nix/temproots/848"5048client # removing stale temporary roots file "/nix/var/nix/temproots/815"5049client # removing stale temporary roots file "/nix/var/nix/temproots/878"5050client # removing stale temporary roots file "/nix/var/nix/temproots/870"5051client # removing stale temporary roots file "/nix/var/nix/temproots/857"5052client # removing stale temporary roots file "/nix/var/nix/temproots/836"5053client # removing stale temporary roots file "/nix/var/nix/temproots/893"5054client # removing stale temporary roots file "/nix/var/nix/temproots/798"5055client # deleting specified paths...5056client # deleting '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top'5057client # deleting unused links...5058client # note: hard linking is currently saving 0.0 KiB5059worker1 # [ 50.124641] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:335905060worker1 # [ 50.131802] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335905061worker1 # [ 50.137764] nix-grpc-daemon[879]: time=2026-09-23T05:12:40.130Z level=INFO msg="All 1 paths already cached"5062worker1 # [ 50.144539] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:335905063worker1 # [ 50.151447] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:335905064worker2 # [ 50.145579] nix-grpc-daemon[783]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:398045065client # copying 3 paths...5066worker1 # [ 50.169691] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:335905067client # copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'grpc://lb:50051'...5068worker1 # [ 50.186422] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:335905069worker1 # [ 50.192122] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=0 paths=1 nar_bytes_out=2325070worker2 # [ 50.179328] nix-grpc-daemon[783]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:398045071worker2 # [ 50.184673] nix-grpc-daemon[783]: ts=2026-09-23T05:12:40Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=0 paths=1 nar_bytes_out=1765072worker2 # [ 50.190645] nix-grpc-daemon[783]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:398045073worker2 # [ 50.194257] nix-grpc-daemon[783]: ts=2026-09-23T05:12:40Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=0 paths=1 nar_bytes_out=1285074client # copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'grpc://lb:50051'...5075client # copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'grpc://lb:50051'...5076client: (finished: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.39 seconds)5077(finished: subtest: outputs read back through any worker carry the cache signature, in 6.19 seconds)5078subtest: two clients wanting the same drv build it once5079client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup5080client # [ 50.308236] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.5081client # Running as unit: dedup1.service; invocation ID: 576c43cadd184c95beeeb0f199bfbe785082client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 0.03 seconds)5083client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup5084worker2 # [ 50.420886] nix-grpc-daemon[783]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:398045085client # [ 50.437694] nix[946]: don't know how to build these paths:5086client # [ 50.438770] nix[946]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5087worker1 # [ 50.448189] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05088worker1 # [ 50.453525] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.3:50051 clients=15089worker1 # [ 50.497318] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335905090worker1 # [ 50.510312] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335905091client # [ 50.518245] nix[946]: copying 1 paths to 'lb:50051'...5092worker1 # [ 50.522499] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:335905093client # [ 50.520391] nix[946]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...5094worker1 # [ 50.535265] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5095worker1 # [ 50.641825] nix-daemon[864]: accepted connection from pid 1059, user nix-grpc-daemon (trusted)5096worker1 # [ 50.655709] nix-daemon[864]: reaped child process 1062, status = succeeded5097lb # [ 50.661459] niks3-server[977]: 2026/09/23 05:12:40 INFO Received uploads request method=POST path=/api/pending_closures5098worker1 # [ 50.671711] nix-grpc-daemon[879]: time=2026-09-23T05:12:40.665Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5099worker1 # [ 50.673193] nix-grpc-daemon[879]: time=2026-09-23T05:12:40.666Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"5100lb # [ 50.696790] niks3-server[977]: 2026/09/23 05:12:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign5101lb # [ 50.700939] niks3-server[977]: 2026/09/23 05:12:40 INFO Signed narinfos id=15 count=15102worker1 # [ 50.707276] nix-grpc-daemon[879]: time=2026-09-23T05:12:40.700Z level=INFO msg="Uploading 1 narinfos"5103lb # [ 50.708268] niks3-server[977]: 2026/09/23 05:12:40 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls5104lb # [ 50.713867] niks3-server[977]: 2026/09/23 05:12:40 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst5105lb # [ 50.727966] niks3-server[977]: 2026/09/23 05:12:40 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete5106lb # [ 50.734198] niks3-server[977]: 2026/09/23 05:12:40 INFO Completed upload id=155107worker1 # [ 50.736479] nix-grpc-daemon[879]: time=2026-09-23T05:12:40.729Z level=INFO msg="Upload complete. (190ms)"5108worker1 # [ 50.742388] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=0 paths=1 nar_bytes_in=4325109worker1 # [ 50.747893] nix-grpc-daemon[780]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335905110worker1 # [ 50.761105] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5111worker1 # [ 50.764136] nix-daemon[864]: reaped child process 1057, status = succeeded5112worker1 # [ 50.769467] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5113client # [ 50.766244] nix[946]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5114lb # [ 50.857691] niks3-server[977]: 2026/09/23 05:12:40 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo5115client # [ 51.359946] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.5116client # Running as unit: dedup2.service; invocation ID: ccde003a771c4378bc6928a3440c5dcb5117client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 1.07 seconds)5118worker2 # [ 51.500666] nix-grpc-daemon[783]: ts=2026-09-23T05:12:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:398045119client # [ 51.519328] nix[962]: don't know how to build these paths:5120client # [ 51.520203] nix[962]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5121worker1 # [ 51.532941] nix-grpc-daemon[780]: ts=2026-09-23T05:12:41Z level=debug event=want client=6 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05122client # [ 51.535925] nix[962]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5123worker1 # [ 51.538804] nix-grpc-daemon[780]: ts=2026-09-23T05:12:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335905124client: must succeed: sleep 25125client: (finished: must succeed: sleep 2, in 2.03 seconds)5126client: waiting for success: ! systemctl is-active dedup1 dedup25127worker1 # [ 54.804230] nix-daemon[864]: reaped child process 1066, status = succeeded5128lb # [ 54.876166] envoy[903]: conn peer=[2001:db8:1::1]:57720 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9537 tx=4378 ms=51255129lb # [ 54.880408] envoy[903]: conn peer=[2001:db8:1::1]:57730 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=410 ms=875130worker1 # [ 54.888086] nix-daemon[864]: accepted connection from pid 1098, user nix-grpc-daemon (trusted)5131lb # [ 54.883985] envoy[903]: conn peer=[2001:db8:1::1]:57726 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1108 tx=2930 ms=1495132lb # [ 54.887518] envoy[903]: conn peer=[2001:db8:1::1]:57752 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=330 ms=735133lb # [ 54.891051] envoy[903]: conn peer=[2001:db8:1::1]:57746 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=408 tx=376 ms=825134worker1 # [ 54.901727] nix-daemon[864]: reaped child process 1101, status = succeeded5135lb # [ 54.907345] niks3-server[977]: 2026/09/23 05:12:44 INFO Received uploads request method=POST path=/api/pending_closures5136worker1 # [ 54.923244] nix-grpc-daemon[879]: time=2026-09-23T05:12:44.916Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5137worker1 # [ 54.926061] nix-grpc-daemon[879]: time=2026-09-23T05:12:44.918Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"5138lb # [ 54.956823] niks3-server[977]: 2026/09/23 05:12:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign5139worker1 # [ 54.966244] nix-grpc-daemon[879]: time=2026-09-23T05:12:44.959Z level=INFO msg="Uploading 1 narinfos"5140lb # [ 54.963548] niks3-server[977]: 2026/09/23 05:12:44 INFO Signed narinfos id=16 count=15141lb # [ 54.973091] niks3-server[977]: 2026/09/23 05:12:44 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete5142lb # [ 54.985871] niks3-server[977]: 2026/09/23 05:12:45 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls5143lb # [ 54.994905] niks3-server[977]: 2026/09/23 05:12:45 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5144lb # [ 55.007321] niks3-server[977]: 2026/09/23 05:12:45 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst5145lb # [ 55.029718] niks3-server[977]: 2026/09/23 05:12:45 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo5146lb # [ 55.041890] niks3-server[977]: 2026/09/23 05:12:45 INFO Completed upload id=165147worker1 # [ 55.044476] nix-grpc-daemon[879]: time=2026-09-23T05:12:45.037Z level=INFO msg="Upload complete. (265ms)"5148worker1 # [ 55.052801] nix-grpc-daemon[780]: ts=2026-09-23T05:12:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=15149worker1 # [ 55.060823] nix-grpc-daemon[780]: ts=2026-09-23T05:12:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=15150worker1 # [ 55.075528] nix-daemon[864]: reaped child process 1064, status = succeeded5151client # [ 55.097586] systemd[1]: dedup1.service: Deactivated successfully.5152client # [ 55.100385] systemd[1]: dedup1.service: Consumed 115ms CPU time over 4.789s wall clock time, 41.5M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic.5153client # [ 55.107517] systemd[1]: dedup2.service: Deactivated successfully.5154client # [ 55.109696] systemd[1]: dedup2.service: Consumed 135ms CPU time over 3.748s wall clock time, 35.4M memory peak, 6.1K incoming IP traffic, 5.7K outgoing IP traffic.5155client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.08 seconds)5156client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }5157client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)5158worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5159worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)5160worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5161worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.03 seconds)5162worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5163worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5164worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5165worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)5166(finished: subtest: two clients wanting the same drv build it once, in 5.69 seconds)5167subtest: --timeout and --max-silent-time reach the builder5168worker1 # [ 56.122835] nix-grpc-daemon[780]: ts=2026-09-23T05:12:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:335905169worker1 # [ 56.137942] nix-grpc-daemon[780]: ts=2026-09-23T05:12:46Z level=debug event=want client=7 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv cached=05170worker2 # [ 56.128237] nix-grpc-daemon[783]: ts=2026-09-23T05:12:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398045171worker1 # [ 56.142948] nix-grpc-daemon[780]: ts=2026-09-23T05:12:46Z level=debug event=assigned drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv worker=192.168.1.4:50051 clients=15172worker2 # [ 56.133911] nix-grpc-daemon[783]: ts=2026-09-23T05:12:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398045173worker2 # [ 56.143465] nix-grpc-daemon[783]: ts=2026-09-23T05:12:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398045174worker2 # [ 56.149940] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5175worker2 # [ 56.242873] nix-daemon[851]: accepted connection from pid 1004, user nix-grpc-daemon (trusted)5176worker2 # [ 56.258054] nix-daemon[851]: reaped child process 1007, status = succeeded5177lb # [ 56.275595] niks3-server[977]: 2026/09/23 05:12:46 INFO Received uploads request method=POST path=/api/pending_closures5178worker2 # [ 56.274106] nix-grpc-daemon[880]: time=2026-09-23T05:12:46.438Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5179worker2 # [ 56.275481] nix-grpc-daemon[880]: time=2026-09-23T05:12:46.438Z level=INFO msg="Uploading qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv (424B)"5180lb # [ 56.306985] niks3-server[977]: 2026/09/23 05:12:46 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.ls5181lb # [ 56.315263] niks3-server[977]: 2026/09/23 05:12:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5182lb # [ 56.321423] niks3-server[977]: 2026/09/23 05:12:46 INFO Signed narinfos id=17 count=15183worker2 # [ 56.312940] nix-grpc-daemon[880]: time=2026-09-23T05:12:46.478Z level=INFO msg="Uploading 1 narinfos"5184lb # [ 56.337315] niks3-server[977]: 2026/09/23 05:12:46 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5185lb # [ 56.448562] niks3-server[977]: 2026/09/23 05:12:46 INFO Registered completed upload object_key=nar/1s4sciw269ksj1s7ymwz5cy4dw9n7bxv3k8wqw69h694spdh4ha2.nar.zst5186lb # [ 56.461207] niks3-server[977]: 2026/09/23 05:12:46 INFO Completed upload id=175187worker2 # [ 56.449954] nix-grpc-daemon[880]: time=2026-09-23T05:12:46.615Z level=INFO msg="Upload complete. (293ms)"5188lb # [ 56.464813] niks3-server[977]: 2026/09/23 05:12:46 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.narinfo5189worker2 # [ 56.455264] nix-grpc-daemon[783]: ts=2026-09-23T05:12:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=0 paths=1 nar_bytes_in=4245190worker2 # [ 56.461558] nix-grpc-daemon[783]: ts=2026-09-23T05:12:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398045191worker2 # [ 56.474161] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5192worker2 # [ 56.483839] nix-daemon[851]: reaped child process 1002, status = succeeded5193worker2 # [ 56.494961] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5194worker2 # [ 62.513783] nix-grpc-daemon[783]: ts=2026-09-23T05:12:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=6 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv assign_id=9 outputs=05195worker2 # [ 62.538890] nix-daemon[851]: reaped child process 1009, status = succeeded5196worker2 # [ 62.541445] nix-daemon[851]: reaped child process 1011, status = succeeded5197worker1 # [ 62.732616] nix-grpc-daemon[780]: ts=2026-09-23T05:12:52Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:335905198worker1 # [ 62.746218] nix-grpc-daemon[780]: ts=2026-09-23T05:12:52Z level=debug event=want client=8 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv cached=05199worker1 # [ 62.751530] nix-grpc-daemon[780]: ts=2026-09-23T05:12:52Z level=debug event=assigned drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv worker=192.168.1.3:50051 clients=15200worker1 # [ 62.757622] nix-grpc-daemon[780]: ts=2026-09-23T05:12:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335905201worker1 # [ 62.769698] nix-grpc-daemon[780]: ts=2026-09-23T05:12:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335905202worker1 # [ 62.781138] nix-grpc-daemon[780]: ts=2026-09-23T05:12:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:335905203worker1 # [ 62.792452] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5204worker1 # [ 62.894789] nix-daemon[864]: accepted connection from pid 1118, user nix-grpc-daemon (trusted)5205worker1 # [ 62.910429] nix-daemon[864]: reaped child process 1121, status = succeeded5206lb # [ 62.914440] niks3-server[977]: 2026/09/23 05:12:52 INFO Received uploads request method=POST path=/api/pending_closures5207worker1 # [ 62.925531] nix-grpc-daemon[879]: time=2026-09-23T05:12:52.918Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5208worker1 # [ 62.927534] nix-grpc-daemon[879]: time=2026-09-23T05:12:52.920Z level=INFO msg="Uploading fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv (424B)"5209lb # [ 62.953662] niks3-server[977]: 2026/09/23 05:12:52 INFO Registered completed upload object_key=nar/013y5sy2r6v0qdwv0dzpgvdfy08bds4mfh2qpvbgjazzrg2v71hp.nar.zst5210lb # [ 62.964813] niks3-server[977]: 2026/09/23 05:12:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5211lb # [ 62.970993] niks3-server[977]: 2026/09/23 05:12:52 INFO Signed narinfos id=18 count=15212worker1 # [ 62.977418] nix-grpc-daemon[879]: time=2026-09-23T05:12:52.970Z level=INFO msg="Uploading 1 narinfos"5213lb # [ 62.983709] niks3-server[977]: 2026/09/23 05:12:53 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5214lb # [ 62.991548] niks3-server[977]: 2026/09/23 05:12:53 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.ls5215lb # [ 62.994835] niks3-server[977]: 2026/09/23 05:12:53 INFO Completed upload id=185216worker1 # [ 62.997721] nix-grpc-daemon[879]: time=2026-09-23T05:12:52.991Z level=INFO msg="Upload complete. (194ms)"5217lb # [ 62.997593] niks3-server[977]: 2026/09/23 05:12:53 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.narinfo5218worker1 # [ 63.003069] nix-grpc-daemon[780]: ts=2026-09-23T05:12:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=0 paths=1 nar_bytes_in=4245219worker1 # [ 63.009521] nix-grpc-daemon[780]: ts=2026-09-23T05:12:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335905220worker1 # [ 63.018881] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5221worker1 # [ 63.028593] nix-daemon[864]: reaped child process 1116, status = succeeded5222worker1 # [ 63.035966] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5223lb # [ 64.876390] envoy[903]: conn peer=[2001:db8:1::1]:57768 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3011 tx=1536 ms=46395224lb # [ 64.880338] envoy[903]: conn peer=[2001:db8:1::1]:57772 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1427 tx=1197 ms=35575225lb # [ 64.884131] envoy[903]: conn peer=[2001:db8:1::1]:60444 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3009 tx=976 ms=64595226worker1 # [ 69.051849] nix-grpc-daemon[780]: ts=2026-09-23T05:12:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=6 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv assign_id=10 outputs=05227worker1 # [ 69.078944] nix-daemon[864]: reaped child process 1123, status = succeeded5228worker1 # [ 69.081458] nix-daemon[864]: reaped child process 1125, status = succeeded5229(finished: subtest: --timeout and --max-silent-time reach the builder, in 13.20 seconds)5230subtest: the first client leaving does not fail the second5231client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share5232client # [ 69.206690] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share.5233client # Running as unit: share1.service; invocation ID: cfeda901a57f4a04ad1cab39673099f35234client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share, in 0.04 seconds)5235client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share5236worker2 # [ 69.326700] nix-grpc-daemon[783]: ts=2026-09-23T05:12:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:398045237client # [ 69.343475] nix[1034]: don't know how to build these paths:5238client # [ 69.344818] nix[1034]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5239worker1 # [ 69.353693] nix-grpc-daemon[780]: ts=2026-09-23T05:12:59Z level=debug event=want client=9 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05240worker1 # [ 69.355608] nix-grpc-daemon[780]: ts=2026-09-23T05:12:59Z level=debug event=assigned drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv worker=192.168.1.4:50051 clients=15241worker2 # [ 69.345746] nix-grpc-daemon[783]: ts=2026-09-23T05:12:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398045242worker2 # [ 69.359380] nix-grpc-daemon[783]: ts=2026-09-23T05:12:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398045243client # [ 69.380068] nix[1034]: copying 1 paths to 'lb:50051'...5244client # [ 69.380901] nix[1034]: copying path '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' to 'lb:50051'...5245worker2 # [ 69.370140] nix-grpc-daemon[783]: ts=2026-09-23T05:12:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398045246worker2 # [ 69.381672] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5247worker2 # [ 69.491429] nix-daemon[851]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)5248worker2 # [ 69.506148] nix-daemon[851]: reaped child process 1037, status = succeeded5249lb # [ 69.526361] niks3-server[977]: 2026/09/23 05:12:59 INFO Received uploads request method=POST path=/api/pending_closures5250worker2 # [ 69.524713] nix-grpc-daemon[880]: time=2026-09-23T05:12:59.689Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5251worker2 # [ 69.526757] nix-grpc-daemon[880]: time=2026-09-23T05:12:59.691Z level=INFO msg="Uploading j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv (432B)"5252lb # [ 69.564248] niks3-server[977]: 2026/09/23 05:12:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5253lb # [ 69.571545] niks3-server[977]: 2026/09/23 05:12:59 INFO Signed narinfos id=19 count=15254worker2 # [ 69.562342] nix-grpc-daemon[880]: time=2026-09-23T05:12:59.727Z level=INFO msg="Uploading 1 narinfos"5255lb # [ 69.579146] niks3-server[977]: 2026/09/23 05:12:59 INFO Registered completed upload object_key=nar/0s0b0gfhn36p0aylk1qa64cgw01bi7a8ca93iv4zg2q7wqcjcqhs.nar.zst5256lb # [ 69.586134] niks3-server[977]: 2026/09/23 05:12:59 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.ls5257lb # [ 69.595497] niks3-server[977]: 2026/09/23 05:12:59 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5258lb # [ 69.598265] niks3-server[977]: 2026/09/23 05:12:59 INFO Completed upload id=195259worker2 # [ 69.586985] nix-grpc-daemon[880]: time=2026-09-23T05:12:59.752Z level=INFO msg="Upload complete. (193ms)"5260lb # [ 69.601725] niks3-server[977]: 2026/09/23 05:12:59 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.narinfo5261worker2 # [ 69.591883] nix-grpc-daemon[783]: ts=2026-09-23T05:12:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=0 paths=1 nar_bytes_in=4325262worker2 # [ 69.597342] nix-grpc-daemon[783]: ts=2026-09-23T05:12:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398045263worker2 # [ 69.609068] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5264worker2 # [ 69.615733] nix-daemon[851]: reaped child process 1032, status = succeeded5265worker2 # [ 69.618466] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5266client # [ 69.629339] nix[1034]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5267client # [ 70.263901] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share.5268client # Running as unit: share2.service; invocation ID: 0997e97631464086913c2bc88ce012595269client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share, in 1.06 seconds)5270client: must succeed: sleep 2; systemctl kill -s INT share15271worker1 # [ 70.408885] nix-grpc-daemon[780]: ts=2026-09-23T05:13:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:335905272client # [ 70.411961] nix[1050]: don't know how to build these paths:5273client # [ 70.412879] nix[1050]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5274worker1 # [ 70.422254] nix-grpc-daemon[780]: ts=2026-09-23T05:13:00Z level=debug event=want client=10 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05275worker2 # [ 70.411260] nix-grpc-daemon[783]: ts=2026-09-23T05:13:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398045276client # [ 70.425201] nix[1050]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5277client # [ 72.413882] systemd[1]: share1.service: Sent signal SIGINT to main process 1034 (nix) on client request.5278client # [ 72.417403] nix[1034]: shutting downj2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5279client # [ 72.420697] nix[1034]: error: interrupted by the user5280client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.04 seconds)5281client: waiting for success: ! systemctl is-active share15282client # [ 72.454852] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE5283client # [ 72.457413] systemd[1]: share1.service: Failed with result 'exit-code'.5284client # [ 72.459463] systemd[1]: share1.service: Consumed 104ms CPU time over 3.252s wall clock time, 41.4M memory peak, 6.1K incoming IP traffic, 8K outgoing IP traffic.5285client: (finished: waiting for success: ! systemctl is-active share1, in 1.07 seconds)5286client: waiting for success: ! systemctl is-active share25287worker2 # [ 73.858232] nix-daemon[851]: reaped child process 1041, status = succeeded5288worker2 # [ 73.939208] nix-daemon[851]: accepted connection from pid 1071, user nix-grpc-daemon (trusted)5289worker2 # [ 73.951786] nix-daemon[851]: reaped child process 1074, status = succeeded5290lb # [ 73.970596] niks3-server[977]: 2026/09/23 05:13:03 INFO Received uploads request method=POST path=/api/pending_closures5291worker2 # [ 73.967453] nix-grpc-daemon[880]: time=2026-09-23T05:13:04.132Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5292worker2 # [ 73.969051] nix-grpc-daemon[880]: time=2026-09-23T05:13:04.134Z level=INFO msg="Uploading b9lg6znd0rinl36gy6ixq1dcj6b8bhhw-slow-share (120B)"5293lb # [ 74.012522] niks3-server[977]: 2026/09/23 05:13:04 INFO Registered completed upload object_key=log/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5294lb # [ 74.027708] niks3-server[977]: 2026/09/23 05:13:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5295worker2 # [ 74.023109] nix-grpc-daemon[880]: time=2026-09-23T05:13:04.188Z level=INFO msg="Uploading 1 narinfos"5296lb # [ 74.035111] niks3-server[977]: 2026/09/23 05:13:04 INFO Signed narinfos id=20 count=15297lb # [ 74.040246] niks3-server[977]: 2026/09/23 05:13:04 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst5298lb # [ 74.048485] niks3-server[977]: 2026/09/23 05:13:04 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.ls5299lb # [ 74.059879] niks3-server[977]: 2026/09/23 05:13:04 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5300lb # [ 74.065884] niks3-server[977]: 2026/09/23 05:13:04 INFO Completed upload id=205301worker2 # [ 74.054770] nix-grpc-daemon[880]: time=2026-09-23T05:13:04.219Z level=INFO msg="Upload complete. (226ms)"5302lb # [ 74.069828] niks3-server[977]: 2026/09/23 05:13:04 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.narinfo5303worker2 # [ 74.062083] nix-grpc-daemon[783]: ts=2026-09-23T05:13:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=3 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15304worker2 # [ 74.069647] nix-grpc-daemon[783]: ts=2026-09-23T05:13:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=4 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15305worker2 # [ 74.083988] nix-daemon[851]: reaped child process 1039, status = succeeded5306client # [ 74.143785] systemd[1]: share2.service: Deactivated successfully.5307client # [ 74.147839] systemd[1]: share2.service: Consumed 114ms CPU time over 3.882s wall clock time, 35.4M memory peak, 6K incoming IP traffic, 5.7K outgoing IP traffic.5308client: (finished: waiting for success: ! systemctl is-active share2, in 1.08 seconds)5309client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }5310client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.03 seconds)5311(finished: subtest: the first client leaving does not fail the second, in 5.61 seconds)5312subtest: build hook: nix-daemon with builders = grpc://lb5313client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log5314client # [ 74.874092] systemd[1]: Started Nix Daemon.5315lb # [ 74.877113] envoy[903]: conn peer=[2001:db8:1::1]:60454 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2965 tx=933 ms=63375316lb # [ 74.880935] envoy[903]: conn peer=[2001:db8:1::1]:49148 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2965 tx=796 ms=30945317lb # [ 74.884456] envoy[903]: conn peer=[2001:db8:1::1]:49152 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1440 tx=1197 ms=37245318client # [ 74.961084] nix-daemon[1102]: accepted connection from pid 1099, user root (trusted)5319worker2 # [ 75.069423] nix-grpc-daemon[783]: ts=2026-09-23T05:13:05Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:398045320worker1 # [ 75.088357] nix-grpc-daemon[780]: ts=2026-09-23T05:13:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335905321worker1 # [ 75.100241] nix-grpc-daemon[780]: ts=2026-09-23T05:13:05Z level=debug event=want client=11 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=05322worker1 # [ 75.105909] nix-grpc-daemon[780]: ts=2026-09-23T05:13:05Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=15323worker1 # [ 75.112856] nix-grpc-daemon[780]: ts=2026-09-23T05:13:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335905324worker1 # [ 75.125475] nix-grpc-daemon[780]: ts=2026-09-23T05:13:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335905325worker1 # [ 75.137369] nix-grpc-daemon[780]: ts=2026-09-23T05:13:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:335905326worker1 # [ 75.148188] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5327worker1 # [ 75.246047] nix-daemon[864]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)5328worker1 # [ 75.259385] nix-daemon[864]: reaped child process 1169, status = succeeded5329lb # [ 75.267942] niks3-server[977]: 2026/09/23 05:13:05 INFO Received uploads request method=POST path=/api/pending_closures5330worker1 # [ 75.277046] nix-grpc-daemon[879]: time=2026-09-23T05:13:05.270Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5331worker1 # [ 75.278990] nix-grpc-daemon[879]: time=2026-09-23T05:13:05.272Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"5332lb # [ 75.296442] niks3-server[977]: 2026/09/23 05:13:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5333worker1 # [ 75.308101] nix-grpc-daemon[879]: time=2026-09-23T05:13:05.301Z level=INFO msg="Uploading 1 narinfos"5334lb # [ 75.305413] niks3-server[977]: 2026/09/23 05:13:05 INFO Signed narinfos id=21 count=15335lb # [ 75.309106] niks3-server[977]: 2026/09/23 05:13:05 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls5336lb # [ 75.315339] niks3-server[977]: 2026/09/23 05:13:05 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst5337lb # [ 75.324062] niks3-server[977]: 2026/09/23 05:13:05 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5338lb # [ 75.328317] niks3-server[977]: 2026/09/23 05:13:05 INFO Completed upload id=215339worker1 # [ 75.330986] nix-grpc-daemon[879]: time=2026-09-23T05:13:05.324Z level=INFO msg="Upload complete. (171ms)"5340lb # [ 75.333121] niks3-server[977]: 2026/09/23 05:13:05 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo5341worker1 # [ 75.335269] nix-grpc-daemon[780]: ts=2026-09-23T05:13:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=0 paths=1 nar_bytes_in=5125342worker1 # [ 75.339820] nix-grpc-daemon[780]: ts=2026-09-23T05:13:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335905343worker1 # [ 75.349141] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5344worker1 # [ 75.354719] nix-daemon[864]: reaped child process 1164, status = succeeded5345worker1 # [ 75.359229] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5346worker1 # [ 76.593241] nix-daemon[864]: reaped child process 1173, status = succeeded5347worker1 # [ 76.678927] nix-daemon[864]: accepted connection from pid 1189, user nix-grpc-daemon (trusted)5348worker1 # [ 76.692959] nix-daemon[864]: reaped child process 1192, status = succeeded5349lb # [ 76.699859] niks3-server[977]: 2026/09/23 05:13:06 INFO Received uploads request method=POST path=/api/pending_closures5350worker1 # [ 76.709335] nix-grpc-daemon[879]: time=2026-09-23T05:13:06.702Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5351worker1 # [ 76.711626] nix-grpc-daemon[879]: time=2026-09-23T05:13:06.704Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"5352lb # [ 76.742499] niks3-server[977]: 2026/09/23 05:13:06 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst5353lb # [ 76.748132] niks3-server[977]: 2026/09/23 05:13:06 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv5354lb # [ 76.753699] niks3-server[977]: 2026/09/23 05:13:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5355lb # [ 76.759138] niks3-server[977]: 2026/09/23 05:13:06 INFO Signed narinfos id=22 count=15356worker1 # [ 76.764539] nix-grpc-daemon[879]: time=2026-09-23T05:13:06.757Z level=INFO msg="Uploading 1 narinfos"5357lb # [ 76.765420] niks3-server[977]: 2026/09/23 05:13:06 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls5358lb # [ 76.772163] niks3-server[977]: 2026/09/23 05:13:06 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5359lb # [ 76.776415] niks3-server[977]: 2026/09/23 05:13:06 INFO Completed upload id=225360worker1 # [ 76.779219] nix-grpc-daemon[879]: time=2026-09-23T05:13:06.772Z level=INFO msg="Upload complete. (209ms)"5361lb # [ 76.778877] niks3-server[977]: 2026/09/23 05:13:06 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo5362worker1 # [ 76.785941] nix-grpc-daemon[780]: ts=2026-09-23T05:13:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=12 outputs=15363worker1 # [ 76.795681] nix-grpc-daemon[780]: ts=2026-09-23T05:13:06Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:335905364worker1 # [ 76.804871] nix-daemon[864]: reaped child process 1171, status = succeeded5365worker2 # [ 76.804385] nix-grpc-daemon[783]: ts=2026-09-23T05:13:06Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:398045366worker2 # [ 76.816386] nix-grpc-daemon[783]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...5367worker2 # [ 76.830362] nix-grpc-daemon[783]: ts=2026-09-23T05:13:06Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=0 paths=1 nar_bytes_out=1285368worker2 # [ 77.036707] nix-grpc-daemon[783]: ts=2026-09-23T05:13:07Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:398045369worker1 # [ 77.054376] nix-grpc-daemon[780]: ts=2026-09-23T05:13:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335905370worker1 # [ 77.063151] nix-grpc-daemon[879]: time=2026-09-23T05:13:07.056Z level=INFO msg="All 1 paths already cached"5371worker1 # [ 77.078276] nix-grpc-daemon[780]: ts=2026-09-23T05:13:07Z level=debug event=want client=12 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=05372worker1 # [ 77.085097] nix-grpc-daemon[780]: ts=2026-09-23T05:13:07Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=15373worker1 # [ 77.090531] nix-grpc-daemon[780]: ts=2026-09-23T05:13:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335905374worker1 # [ 77.098456] nix-grpc-daemon[780]: ts=2026-09-23T05:13:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335905375worker1 # [ 77.110786] nix-grpc-daemon[879]: time=2026-09-23T05:13:07.104Z level=INFO msg="All 1 paths already cached"5376worker1 # [ 77.118720] nix-grpc-daemon[780]: ts=2026-09-23T05:13:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:335905377worker1 # [ 77.128353] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5378worker1 # [ 77.226934] nix-daemon[864]: accepted connection from pid 1198, user nix-grpc-daemon (trusted)5379worker1 # [ 77.241103] nix-daemon[864]: reaped child process 1201, status = succeeded5380lb # [ 77.245245] niks3-server[977]: 2026/09/23 05:13:07 INFO Received uploads request method=POST path=/api/pending_closures5381worker1 # [ 77.257547] nix-grpc-daemon[879]: time=2026-09-23T05:13:07.251Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5382worker1 # [ 77.259612] nix-grpc-daemon[879]: time=2026-09-23T05:13:07.252Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"5383lb # [ 77.282056] niks3-server[977]: 2026/09/23 05:13:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5384lb # [ 77.288194] niks3-server[977]: 2026/09/23 05:13:07 INFO Signed narinfos id=23 count=15385worker1 # [ 77.293249] nix-grpc-daemon[879]: time=2026-09-23T05:13:07.286Z level=INFO msg="Uploading 1 narinfos"5386lb # [ 77.293273] niks3-server[977]: 2026/09/23 05:13:07 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst5387lb # [ 77.298107] niks3-server[977]: 2026/09/23 05:13:07 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls5388lb # [ 77.310199] niks3-server[977]: 2026/09/23 05:13:07 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5389lb # [ 77.313411] niks3-server[977]: 2026/09/23 05:13:07 INFO Completed upload id=235390worker1 # [ 77.316244] nix-grpc-daemon[879]: time=2026-09-23T05:13:07.309Z level=INFO msg="Upload complete. (175ms)"5391lb # [ 77.315904] niks3-server[977]: 2026/09/23 05:13:07 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo5392worker1 # [ 77.320579] nix-grpc-daemon[780]: ts=2026-09-23T05:13:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=0 paths=1 nar_bytes_in=6245393worker1 # [ 77.326235] nix-grpc-daemon[780]: ts=2026-09-23T05:13:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335905394worker1 # [ 77.338908] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5395worker1 # [ 77.343159] nix-daemon[864]: reaped child process 1196, status = succeeded5396worker1 # [ 77.346791] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5397worker1 # [ 78.390301] nix-daemon[864]: reaped child process 1205, status = succeeded5398worker1 # [ 78.468942] nix-daemon[864]: accepted connection from pid 1221, user nix-grpc-daemon (trusted)5399worker1 # [ 78.481644] nix-daemon[864]: reaped child process 1224, status = succeeded5400lb # [ 78.487781] niks3-server[977]: 2026/09/23 05:13:08 INFO Received uploads request method=POST path=/api/pending_closures5401worker1 # [ 78.497273] nix-grpc-daemon[879]: time=2026-09-23T05:13:08.490Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5402worker1 # [ 78.498879] nix-grpc-daemon[879]: time=2026-09-23T05:13:08.492Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"5403lb # [ 78.531052] niks3-server[977]: 2026/09/23 05:13:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5404lb # [ 78.544725] niks3-server[977]: 2026/09/23 05:13:08 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls5405worker1 # [ 78.551611] nix-grpc-daemon[879]: time=2026-09-23T05:13:08.544Z level=INFO msg="Uploading 1 narinfos"5406lb # [ 78.548783] niks3-server[977]: 2026/09/23 05:13:08 INFO Signed narinfos id=24 count=15407lb # [ 78.553066] niks3-server[977]: 2026/09/23 05:13:08 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv5408lb # [ 78.557790] niks3-server[977]: 2026/09/23 05:13:08 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst5409lb # [ 78.568944] niks3-server[977]: 2026/09/23 05:13:08 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5410lb # [ 78.571360] niks3-server[977]: 2026/09/23 05:13:08 INFO Completed upload id=245411worker1 # [ 78.574152] nix-grpc-daemon[879]: time=2026-09-23T05:13:08.567Z level=INFO msg="Upload complete. (219ms)"5412lb # [ 78.574096] niks3-server[977]: 2026/09/23 05:13:08 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo5413worker1 # [ 78.580946] nix-grpc-daemon[780]: ts=2026-09-23T05:13:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=13 outputs=15414worker2 # [ 78.577318] nix-grpc-daemon[783]: ts=2026-09-23T05:13:08Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:398045415worker1 # [ 78.596813] nix-daemon[864]: reaped child process 1203, status = succeeded5416worker2 # [ 78.586259] nix-grpc-daemon[783]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...5417worker2 # [ 78.611892] nix-grpc-daemon[783]: ts=2026-09-23T05:13:08Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:398045418worker2 # [ 78.617508] nix-grpc-daemon[783]: ts=2026-09-23T05:13:08Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=0 paths=1 nar_bytes_out=1765419worker2 # [ 78.819492] nix-grpc-daemon[783]: ts=2026-09-23T05:13:08Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:398045420worker2 # [ 78.825799] nix-grpc-daemon[783]: ts=2026-09-23T05:13:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398045421worker2 # [ 78.831644] nix-grpc-daemon[880]: time=2026-09-23T05:13:08.995Z level=INFO msg="All 2 paths already cached"5422worker1 # [ 78.856488] nix-grpc-daemon[780]: ts=2026-09-23T05:13:08Z level=debug event=want client=13 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05423worker1 # [ 78.862239] nix-grpc-daemon[780]: ts=2026-09-23T05:13:08Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=15424worker1 # [ 78.867895] nix-grpc-daemon[780]: ts=2026-09-23T05:13:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335905425worker1 # [ 78.880275] nix-grpc-daemon[780]: ts=2026-09-23T05:13:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335905426worker1 # [ 78.894814] nix-grpc-daemon[879]: time=2026-09-23T05:13:08.888Z level=INFO msg="All 2 paths already cached"5427worker1 # [ 78.903520] nix-grpc-daemon[780]: ts=2026-09-23T05:13:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:335905428worker1 # [ 78.914132] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5429worker1 # [ 79.017436] nix-daemon[864]: accepted connection from pid 1230, user nix-grpc-daemon (trusted)5430worker1 # [ 79.032454] nix-daemon[864]: reaped child process 1233, status = succeeded5431lb # [ 79.036648] niks3-server[977]: 2026/09/23 05:13:09 INFO Received uploads request method=POST path=/api/pending_closures5432worker1 # [ 79.047669] nix-grpc-daemon[879]: time=2026-09-23T05:13:09.041Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5433worker1 # [ 79.049162] nix-grpc-daemon[879]: time=2026-09-23T05:13:09.042Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5434lb # [ 79.068713] niks3-server[977]: 2026/09/23 05:13:09 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5435lb # [ 79.078393] niks3-server[977]: 2026/09/23 05:13:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5436lb # [ 79.087718] niks3-server[977]: 2026/09/23 05:13:09 INFO Signed narinfos id=25 count=15437worker1 # [ 79.093408] nix-grpc-daemon[879]: time=2026-09-23T05:13:09.086Z level=INFO msg="Uploading 1 narinfos"5438lb # [ 79.095356] niks3-server[977]: 2026/09/23 05:13:09 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5439lb # [ 79.109369] niks3-server[977]: 2026/09/23 05:13:09 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5440lb # [ 79.116739] niks3-server[977]: 2026/09/23 05:13:09 INFO Completed upload id=255441worker1 # [ 79.119537] nix-grpc-daemon[879]: time=2026-09-23T05:13:09.112Z level=INFO msg="Upload complete. (192ms)"5442lb # [ 79.118794] niks3-server[977]: 2026/09/23 05:13:09 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo5443worker1 # [ 79.123914] nix-grpc-daemon[780]: ts=2026-09-23T05:13:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=0 paths=1 nar_bytes_in=7525444worker1 # [ 79.130111] nix-grpc-daemon[780]: ts=2026-09-23T05:13:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335905445worker1 # [ 79.143114] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5446worker1 # [ 79.150194] nix-daemon[864]: reaped child process 1228, status = succeeded5447worker1 # [ 79.158346] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5448worker1 # [ 79.974559] nix-daemon[864]: reaped child process 1237, status = succeeded5449worker1 # [ 80.048701] nix-daemon[864]: accepted connection from pid 1253, user nix-grpc-daemon (trusted)5450worker1 # [ 80.062153] nix-daemon[864]: reaped child process 1256, status = succeeded5451lb # [ 80.068521] niks3-server[977]: 2026/09/23 05:13:10 INFO Received uploads request method=POST path=/api/pending_closures5452worker1 # [ 80.082335] nix-grpc-daemon[879]: time=2026-09-23T05:13:10.075Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5453worker1 # [ 80.083815] nix-grpc-daemon[879]: time=2026-09-23T05:13:10.077Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5454lb # [ 80.109425] niks3-server[977]: 2026/09/23 05:13:10 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5455lb # [ 80.122117] niks3-server[977]: 2026/09/23 05:13:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5456lb # [ 80.128245] niks3-server[977]: 2026/09/23 05:13:10 INFO Signed narinfos id=26 count=15457worker1 # [ 80.134327] nix-grpc-daemon[879]: time=2026-09-23T05:13:10.127Z level=INFO msg="Uploading 1 narinfos"5458lb # [ 80.138107] niks3-server[977]: 2026/09/23 05:13:10 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5459lb # [ 80.148745] niks3-server[977]: 2026/09/23 05:13:10 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5460lb # [ 80.151947] niks3-server[977]: 2026/09/23 05:13:10 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5461lb # [ 80.153728] niks3-server[977]: 2026/09/23 05:13:10 INFO Completed upload id=265462worker1 # [ 80.158297] nix-grpc-daemon[879]: time=2026-09-23T05:13:10.151Z level=INFO msg="Upload complete. (193ms)"5463lb # [ 80.158130] niks3-server[977]: 2026/09/23 05:13:10 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5464worker1 # [ 80.165493] nix-grpc-daemon[780]: ts=2026-09-23T05:13:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=14 outputs=15465worker2 # [ 80.162234] nix-grpc-daemon[783]: ts=2026-09-23T05:13:10Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:398045466worker1 # [ 80.181192] nix-daemon[864]: reaped child process 1235, status = succeeded5467worker2 # [ 80.171680] nix-grpc-daemon[783]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...5468worker1 # [ 80.209723] nix-grpc-daemon[780]: ts=2026-09-23T05:13:10Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:335905469worker1 # [ 80.215768] nix-grpc-daemon[780]: ts=2026-09-23T05:13:10Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=0 paths=1 nar_bytes_out=2325470client # [ 80.281319] nix-daemon[1102]: reaped child process 1104, status = succeeded5471client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log, in 5.51 seconds)5472client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5473client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.02 seconds)5474client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5475client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.02 seconds)5476client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5477client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.02 seconds)5478(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 5.56 seconds)5479subtest: an input only one worker has still reaches the builder5480client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only5481client # this derivation will be built:5482client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5483client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5484client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only, in 0.23 seconds)5485client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5486client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.09 seconds)5487worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5488worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.12 seconds)5489worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5490worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.02 seconds)5491client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&25492client # [ 80.934730] nix-daemon[1102]: accepted connection from pid 1185, user root (trusted)5493client # this derivation will be built:5494client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5495worker1 # [ 81.053960] nix-grpc-daemon[780]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:335905496client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5497worker2 # [ 81.043684] nix-grpc-daemon[783]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398045498client # copying 1 paths...5499client # copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' to 'grpc://lb:50051'...5500worker1 # [ 81.070811] nix-grpc-daemon[780]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:335905501worker1 # [ 81.081990] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5502worker1 # [ 81.194788] nix-daemon[864]: accepted connection from pid 1265, user nix-grpc-daemon (trusted)5503worker1 # [ 81.208286] nix-daemon[864]: reaped child process 1268, status = succeeded5504lb # [ 81.215832] niks3-server[977]: 2026/09/23 05:13:11 INFO Received uploads request method=POST path=/api/pending_closures5505worker1 # [ 81.224983] nix-grpc-daemon[879]: time=2026-09-23T05:13:11.218Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5506worker1 # [ 81.226445] nix-grpc-daemon[879]: time=2026-09-23T05:13:11.220Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5507lb # [ 81.248147] niks3-server[977]: 2026/09/23 05:13:11 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5508lb # [ 81.256514] niks3-server[977]: 2026/09/23 05:13:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5509lb # [ 81.262893] niks3-server[977]: 2026/09/23 05:13:11 INFO Signed narinfos id=27 count=15510worker1 # [ 81.268608] nix-grpc-daemon[879]: time=2026-09-23T05:13:11.261Z level=INFO msg="Uploading 1 narinfos"5511lb # [ 81.270536] niks3-server[977]: 2026/09/23 05:13:11 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5512lb # [ 81.281162] niks3-server[977]: 2026/09/23 05:13:11 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5513lb # [ 81.289221] niks3-server[977]: 2026/09/23 05:13:11 INFO Completed upload id=275514worker1 # [ 81.291601] nix-grpc-daemon[879]: time=2026-09-23T05:13:11.285Z level=INFO msg="Upload complete. (199ms)"5515lb # [ 81.291505] niks3-server[977]: 2026/09/23 05:13:11 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5516worker1 # [ 81.296651] nix-grpc-daemon[780]: ts=2026-09-23T05:13:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=0 paths=1 nar_bytes_in=1205517worker1 # [ 81.310193] nix-grpc-daemon[780]: ts=2026-09-23T05:13:11Z level=debug event=want client=14 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05518worker1 # [ 81.313696] nix-grpc-daemon[780]: ts=2026-09-23T05:13:11Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=15519worker1 # [ 81.321191] nix-daemon[864]: reaped child process 1263, status = succeeded5520worker2 # [ 81.341392] nix-grpc-daemon[783]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398045521worker2 # [ 81.354385] nix-grpc-daemon[783]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398045522client # copying 2 paths to 'lb:50051'...5523client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5524worker2 # [ 81.370555] nix-grpc-daemon[783]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398045525client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5526worker2 # [ 81.381685] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5527worker2 # [ 81.491934] nix-daemon[851]: accepted connection from pid 1084, user nix-grpc-daemon (trusted)5528worker2 # [ 81.506045] nix-daemon[851]: reaped child process 1087, status = succeeded5529lb # [ 81.523579] niks3-server[977]: 2026/09/23 05:13:11 INFO Received uploads request method=POST path=/api/pending_closures5530lb # [ 81.532544] niks3-server[977]: 2026/09/23 05:13:11 INFO Received uploads request method=POST path=/api/pending_closures5531worker2 # [ 81.530079] nix-grpc-daemon[880]: time=2026-09-23T05:13:11.695Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5532worker2 # [ 81.533209] nix-grpc-daemon[880]: time=2026-09-23T05:13:11.696Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5533worker2 # [ 81.536242] nix-grpc-daemon[880]: time=2026-09-23T05:13:11.697Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5534lb # [ 81.570542] niks3-server[977]: 2026/09/23 05:13:11 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5535lb # [ 81.580524] niks3-server[977]: 2026/09/23 05:13:11 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5536lb # [ 81.593261] niks3-server[977]: 2026/09/23 05:13:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5537lb # [ 81.596056] niks3-server[977]: 2026/09/23 05:13:11 INFO Signed narinfos id=28 count=15538lb # [ 81.597658] niks3-server[977]: 2026/09/23 05:13:11 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5539lb # [ 81.599890] niks3-server[977]: 2026/09/23 05:13:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5540lb # [ 81.602043] niks3-server[977]: 2026/09/23 05:13:11 INFO Signed narinfos id=29 count=25541worker2 # [ 81.592646] nix-grpc-daemon[880]: time=2026-09-23T05:13:11.757Z level=INFO msg="Uploading 3 narinfos"5542lb # [ 81.606323] niks3-server[977]: 2026/09/23 05:13:11 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5543lb # [ 81.616171] niks3-server[977]: 2026/09/23 05:13:11 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5544lb # [ 81.619930] niks3-server[977]: 2026/09/23 05:13:11 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5545lb # [ 81.623564] niks3-server[977]: 2026/09/23 05:13:11 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5546lb # [ 81.625363] niks3-server[977]: 2026/09/23 05:13:11 INFO Completed upload id=285547lb # [ 81.628119] niks3-server[977]: 2026/09/23 05:13:11 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5548lb # [ 81.630444] niks3-server[977]: 2026/09/23 05:13:11 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5549lb # [ 81.633305] niks3-server[977]: 2026/09/23 05:13:11 INFO Completed upload id=295550worker2 # [ 81.621929] nix-grpc-daemon[880]: time=2026-09-23T05:13:11.787Z level=INFO msg="Upload complete. (225ms)"5551worker2 # [ 81.626961] nix-grpc-daemon[783]: ts=2026-09-23T05:13:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=0 paths=2 nar_bytes_in=9765552worker2 # [ 81.632608] nix-grpc-daemon[783]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398045553worker2 # [ 81.645264] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5554worker2 # [ 81.654235] nix-daemon[851]: reaped child process 1082, status = succeeded5555worker2 # [ 81.664375] nix-grpc-daemon[783]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5556client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5557worker2 # [ 81.678414] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5558client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5559client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5560worker2 # [ 82.915791] nix-daemon[851]: reaped child process 1092, status = succeeded5561worker2 # [ 82.997458] nix-daemon[851]: accepted connection from pid 1108, user nix-grpc-daemon (trusted)5562worker2 # [ 83.010330] nix-daemon[851]: reaped child process 1111, status = succeeded5563lb # [ 83.029795] niks3-server[977]: 2026/09/23 05:13:13 INFO Received uploads request method=POST path=/api/pending_closures5564worker2 # [ 83.027612] nix-grpc-daemon[880]: time=2026-09-23T05:13:13.192Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5565lb # [ 83.061610] niks3-server[977]: 2026/09/23 05:13:13 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5566lb # [ 83.069227] niks3-server[977]: 2026/09/23 05:13:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5567worker2 # [ 83.062853] nix-grpc-daemon[880]: time=2026-09-23T05:13:13.228Z level=INFO msg="Uploading 1 narinfos"5568lb # [ 83.075199] niks3-server[977]: 2026/09/23 05:13:13 INFO Signed narinfos id=30 count=15569lb # [ 83.081425] niks3-server[977]: 2026/09/23 05:13:13 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5570lb # [ 83.094737] niks3-server[977]: 2026/09/23 05:13:13 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5571lb # [ 83.101346] niks3-server[977]: 2026/09/23 05:13:13 INFO Completed upload id=305572worker2 # [ 83.089857] nix-grpc-daemon[880]: time=2026-09-23T05:13:13.255Z level=INFO msg="Upload complete. (203ms)"5573lb # [ 83.106933] niks3-server[977]: 2026/09/23 05:13:13 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5574client # copying 1 paths...5575worker2 # [ 83.094698] nix-grpc-daemon[783]: ts=2026-09-23T05:13:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=15 outputs=15576worker1 # [ 83.115805] nix-grpc-daemon[780]: ts=2026-09-23T05:13:13Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:335905577worker2 # [ 83.107821] nix-daemon[851]: reaped child process 1089, status = succeeded5578worker1 # [ 83.124725] nix-grpc-daemon[780]: copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'http://lb:5751'...5579client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5580worker2 # [ 83.136914] nix-grpc-daemon[783]: ts=2026-09-23T05:13:13Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:398045581worker2 # [ 83.142140] nix-grpc-daemon[783]: ts=2026-09-23T05:13:13Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=0 paths=1 nar_bytes_out=1205582client # [ 83.223439] nix-daemon[1102]: reaped child process 1188, status = succeeded5583client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.41 seconds)5584client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&25585client # [ 83.324246] nix-daemon[1102]: accepted connection from pid 1207, user root (trusted)5586client # this derivation will be built:5587client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5588worker1 # [ 83.444151] nix-grpc-daemon[780]: ts=2026-09-23T05:13:13Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:335905589client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5590worker2 # [ 83.438184] nix-grpc-daemon[783]: ts=2026-09-23T05:13:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398045591worker2 # [ 83.448063] nix-grpc-daemon[880]: time=2026-09-23T05:13:13.613Z level=INFO msg="All 1 paths already cached"5592worker1 # [ 83.476758] nix-grpc-daemon[780]: ts=2026-09-23T05:13:13Z level=debug event=want client=15 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05593worker1 # [ 83.483470] nix-grpc-daemon[780]: ts=2026-09-23T05:13:13Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15594worker1 # [ 83.488661] nix-grpc-daemon[780]: ts=2026-09-23T05:13:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335905595worker1 # [ 83.500259] nix-grpc-daemon[780]: ts=2026-09-23T05:13:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335905596client # copying 1 paths to 'lb:50051'...5597client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5598worker1 # [ 83.518874] nix-grpc-daemon[780]: ts=2026-09-23T05:13:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:335905599worker1 # [ 83.530184] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5600worker1 # [ 83.545747] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 7 ms (attempt 1/5)5601worker1 # [ 83.558550] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 39 ms (attempt 2/5)5602worker1 # [ 83.604134] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 65 ms (attempt 3/5)5603worker1 # [ 83.671247] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 425 ms (attempt 4/5)5604worker1 # [ 84.098767] nix-grpc-daemon[780]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5605worker1 # [ 84.102100] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5606worker1 # [ 84.108092] nix-grpc-daemon[780]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5607worker1 # [ 84.227588] nix-daemon[864]: accepted connection from pid 1282, user nix-grpc-daemon (trusted)5608worker1 # [ 84.242457] nix-daemon[864]: reaped child process 1285, status = succeeded5609lb # [ 84.248671] niks3-server[977]: 2026/09/23 05:13:14 INFO Received uploads request method=POST path=/api/pending_closures5610worker1 # [ 84.259703] nix-grpc-daemon[879]: time=2026-09-23T05:13:14.253Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5611worker1 # [ 84.261320] nix-grpc-daemon[879]: time=2026-09-23T05:13:14.254Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5612lb # [ 84.282055] niks3-server[977]: 2026/09/23 05:13:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5613lb # [ 84.290135] niks3-server[977]: 2026/09/23 05:13:14 INFO Signed narinfos id=31 count=15614worker1 # [ 84.296531] nix-grpc-daemon[879]: time=2026-09-23T05:13:14.289Z level=INFO msg="Uploading 1 narinfos"5615lb # [ 84.296178] niks3-server[977]: 2026/09/23 05:13:14 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5616lb # [ 84.303076] niks3-server[977]: 2026/09/23 05:13:14 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5617lb # [ 84.314613] niks3-server[977]: 2026/09/23 05:13:14 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5618lb # [ 84.322416] niks3-server[977]: 2026/09/23 05:13:14 INFO Completed upload id=315619worker1 # [ 84.324636] nix-grpc-daemon[879]: time=2026-09-23T05:13:14.318Z level=INFO msg="Upload complete. (197ms)"5620lb # [ 84.325423] niks3-server[977]: 2026/09/23 05:13:14 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5621worker1 # [ 84.330102] nix-grpc-daemon[780]: ts=2026-09-23T05:13:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=0 paths=1 nar_bytes_in=5605622worker1 # [ 84.334980] nix-grpc-daemon[780]: ts=2026-09-23T05:13:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:335905623worker1 # [ 84.346111] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5624worker1 # [ 84.350542] nix-daemon[864]: reaped child process 1275, status = succeeded5625worker1 # [ 84.355738] nix-daemon[864]: accepted connection from pid 780, user nix-grpc-daemon (trusted)5626client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5627client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5628lb # [ 84.877982] envoy[903]: conn peer=[2001:db8:1::1]:40960 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3409 tx=1919 ms=18165629lb # [ 84.881862] envoy[903]: conn peer=[2001:db8:1::1]:40972 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=334 ms=855630lb # [ 84.885356] envoy[903]: conn peer=[2001:db8:1::1]:40982 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3863 tx=2137 ms=16335631lb # [ 84.888840] envoy[903]: conn peer=[2001:db8:1::1]:40996 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=585632lb # [ 84.892535] envoy[903]: conn peer=[2001:db8:1::1]:41000 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4324 tx=2371 ms=13935633lb # [ 84.896467] envoy[903]: conn peer=[2001:db8:1::1]:41004 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=410 ms=175634lb # [ 84.899947] envoy[903]: conn peer=[2001:db8:1::1]:41018 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4220 tx=2054 ms=21125635lb # [ 84.904191] envoy[903]: conn peer=[2001:db8:1::1]:43674 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=165636client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5637worker1 # [ 85.391870] nix-daemon[864]: reaped child process 1289, status = succeeded5638worker1 # [ 85.479764] nix-daemon[864]: accepted connection from pid 1305, user nix-grpc-daemon (trusted)5639worker1 # [ 85.491616] nix-daemon[864]: reaped child process 1308, status = succeeded5640lb # [ 85.498709] niks3-server[977]: 2026/09/23 05:13:15 INFO Received uploads request method=POST path=/api/pending_closures5641worker1 # [ 85.510251] nix-grpc-daemon[879]: time=2026-09-23T05:13:15.503Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5642lb # [ 85.532352] niks3-server[977]: 2026/09/23 05:13:15 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5643lb # [ 85.538967] niks3-server[977]: 2026/09/23 05:13:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5644lb # [ 85.544065] niks3-server[977]: 2026/09/23 05:13:15 INFO Signed narinfos id=32 count=15645worker1 # [ 85.549653] nix-grpc-daemon[879]: time=2026-09-23T05:13:15.543Z level=INFO msg="Uploading 1 narinfos"5646lb # [ 85.554147] niks3-server[977]: 2026/09/23 05:13:15 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5647lb # [ 85.565108] niks3-server[977]: 2026/09/23 05:13:15 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5648lb # [ 85.570220] niks3-server[977]: 2026/09/23 05:13:15 INFO Completed upload id=325649worker1 # [ 85.572925] nix-grpc-daemon[879]: time=2026-09-23T05:13:15.566Z level=INFO msg="Upload complete. (209ms)"5650lb # [ 85.575593] niks3-server[977]: 2026/09/23 05:13:15 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5651worker1 # [ 85.578887] nix-grpc-daemon[780]: ts=2026-09-23T05:13:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=16 outputs=15652client # copying 1 paths...5653worker2 # [ 85.576113] nix-grpc-daemon[783]: ts=2026-09-23T05:13:15Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:398045654worker1 # [ 85.591688] nix-daemon[864]: reaped child process 1287, status = succeeded5655worker2 # [ 85.584811] nix-grpc-daemon[783]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5656client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5657worker1 # [ 85.624675] nix-grpc-daemon[780]: ts=2026-09-23T05:13:15Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:335905658worker1 # [ 85.629850] nix-grpc-daemon[780]: ts=2026-09-23T05:13:15Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:33590 duration_s=0 paths=1 nar_bytes_out=1205659client # [ 85.693265] nix-daemon[1102]: reaped child process 1210, status = succeeded5660client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&2, in 2.47 seconds)5661client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&25662client # [ 85.799658] nix-daemon[1102]: accepted connection from pid 1229, user root (trusted)5663client # this derivation will be built:5664client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5665worker2 # [ 85.902712] nix-grpc-daemon[783]: ts=2026-09-23T05:13:16Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:398045666client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5667worker1 # [ 85.924465] nix-grpc-daemon[780]: ts=2026-09-23T05:13:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:335905668worker1 # [ 85.934475] nix-grpc-daemon[879]: time=2026-09-23T05:13:15.927Z level=INFO msg="All 1 paths already cached"5669worker1 # [ 85.949166] nix-grpc-daemon[780]: ts=2026-09-23T05:13:15Z level=debug event=want client=16 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05670worker1 # [ 85.953615] nix-grpc-daemon[780]: ts=2026-09-23T05:13:15Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15671worker2 # [ 85.947196] nix-grpc-daemon[783]: ts=2026-09-23T05:13:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398045672worker2 # [ 85.959078] nix-grpc-daemon[783]: ts=2026-09-23T05:13:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:398045673worker2 # [ 85.972588] nix-grpc-daemon[880]: time=2026-09-23T05:13:16.137Z level=INFO msg="All 1 paths already cached"5674client # copying 1 paths to 'lb:50051'...5675client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5676worker2 # [ 85.981351] nix-grpc-daemon[783]: ts=2026-09-23T05:13:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:398045677worker2 # [ 85.992390] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5678worker2 # [ 86.091747] nix-daemon[851]: accepted connection from pid 1116, user nix-grpc-daemon (trusted)5679worker2 # [ 86.104600] nix-daemon[851]: reaped child process 1119, status = succeeded5680lb # [ 86.123546] niks3-server[977]: 2026/09/23 05:13:16 INFO Received uploads request method=POST path=/api/pending_closures5681worker2 # [ 86.122221] nix-grpc-daemon[880]: time=2026-09-23T05:13:16.287Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5682worker2 # [ 86.124173] nix-grpc-daemon[880]: time=2026-09-23T05:13:16.289Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5683lb # [ 86.157244] niks3-server[977]: 2026/09/23 05:13:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5684worker2 # [ 86.151300] nix-grpc-daemon[880]: time=2026-09-23T05:13:16.316Z level=INFO msg="Uploading 1 narinfos"5685lb # [ 86.162748] niks3-server[977]: 2026/09/23 05:13:16 INFO Signed narinfos id=33 count=15686lb # [ 86.171094] niks3-server[977]: 2026/09/23 05:13:16 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5687lb # [ 86.181098] niks3-server[977]: 2026/09/23 05:13:16 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5688lb # [ 86.186802] niks3-server[977]: 2026/09/23 05:13:16 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5689lb # [ 86.190434] niks3-server[977]: 2026/09/23 05:13:16 INFO Completed upload id=335690worker2 # [ 86.179069] nix-grpc-daemon[880]: time=2026-09-23T05:13:16.344Z level=INFO msg="Upload complete. (175ms)"5691lb # [ 86.192196] niks3-server[977]: 2026/09/23 05:13:16 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5692worker2 # [ 86.183066] nix-grpc-daemon[783]: ts=2026-09-23T05:13:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=0 paths=1 nar_bytes_in=5605693worker2 # [ 86.188303] nix-grpc-daemon[783]: ts=2026-09-23T05:13:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:398045694worker2 # [ 86.199327] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5695worker2 # [ 86.200728] nix-daemon[851]: reaped child process 1114, status = succeeded5696worker2 # [ 86.205882] nix-daemon[851]: accepted connection from pid 783, user nix-grpc-daemon (trusted)5697client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5698client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5699client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5700worker2 # [ 87.640295] nix-daemon[851]: reaped child process 1123, status = succeeded5701worker2 # [ 87.725363] nix-daemon[851]: accepted connection from pid 1139, user nix-grpc-daemon (trusted)5702worker2 # [ 87.736939] nix-daemon[851]: reaped child process 1142, status = succeeded5703lb # [ 87.757998] niks3-server[977]: 2026/09/23 05:13:17 INFO Received uploads request method=POST path=/api/pending_closures5704worker2 # [ 87.752865] nix-grpc-daemon[880]: time=2026-09-23T05:13:17.918Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5705lb # [ 87.786310] niks3-server[977]: 2026/09/23 05:13:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5706lb # [ 87.793632] niks3-server[977]: 2026/09/23 05:13:17 INFO Signed narinfos id=34 count=15707worker2 # [ 87.785394] nix-grpc-daemon[880]: time=2026-09-23T05:13:17.950Z level=INFO msg="Uploading 1 narinfos"5708lb # [ 87.804772] niks3-server[977]: 2026/09/23 05:13:17 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5709lb # [ 87.810914] niks3-server[977]: 2026/09/23 05:13:17 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5710lb # [ 87.820111] niks3-server[977]: 2026/09/23 05:13:17 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5711lb # [ 87.825274] niks3-server[977]: 2026/09/23 05:13:17 INFO Completed upload id=345712worker2 # [ 87.816495] nix-grpc-daemon[880]: time=2026-09-23T05:13:17.981Z level=INFO msg="Upload complete. (200ms)"5713lb # [ 87.830937] niks3-server[977]: 2026/09/23 05:13:17 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5714worker2 # [ 87.822665] nix-grpc-daemon[783]: ts=2026-09-23T05:13:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=17 outputs=15715worker2 # [ 87.836572] nix-daemon[851]: reaped child process 1121, status = succeeded5716client # copying 1 paths...5717worker1 # [ 87.884499] nix-grpc-daemon[780]: ts=2026-09-23T05:13:17Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:335905718worker1 # [ 87.896876] nix-grpc-daemon[780]: copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'http://lb:5751'...5719client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5720worker2 # [ 87.916434] nix-grpc-daemon[783]: ts=2026-09-23T05:13:18Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:398045721worker2 # [ 87.921114] nix-grpc-daemon[783]: ts=2026-09-23T05:13:18Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:39804 duration_s=0 paths=1 nar_bytes_out=1205722client # [ 87.996382] nix-daemon[1102]: reaped child process 1232, status = succeeded5723client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.30 seconds)5724(finished: subtest: an input only one worker has still reaches the builder, in 7.65 seconds)5725subtest: upload whose reference the worker lacks is completed from the cache5726client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache5727client # this derivation will be built:5728client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5729client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5730client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache, in 0.24 seconds)5731client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache5732client # this derivation will be built:5733client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5734client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5735client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache, in 0.23 seconds)5736client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5737client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.10 seconds)5738worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5739worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.13 seconds)5740worker2 # [ 88.731918] systemd[1]: Stopping Nix worker-protocol over gRPC...5741worker2 # [ 89.665220] nix-grpc-daemon[783]: ts=2026-09-23T05:13:19Z level=info event=signal_exit5742worker1 # [ 90.715192] nix-grpc-daemon[780]: ts=2026-09-23T05:13:20Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:566485743worker1 # [ 90.718817] nix-grpc-daemon[780]: ts=2026-09-23T05:13:20Z level=info event=worker_gone addr=192.168.1.4:500515744worker2 # [ 90.705870] nix-grpc-daemon[783]: ts=2026-09-23T05:13:20Z level=info event=scheduler_restarting5745worker2 # [ 90.717674] nix-grpc-daemon[783]: ts=2026-09-23T05:13:20Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5746worker2 # [ 90.740533] nix-daemon[851]: reaped child process 853, status = succeeded5747worker2 # [ 90.803355] nix-grpc-daemon[783]: E0000 00:00:1790140400.968374 829 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true5748worker2 # [ 90.808638] nix-grpc-daemon[783]: E0000 00:00:1790140400.973523 829 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5749worker2 # [ 92.026732] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5750worker2 # [ 92.030724] systemd[1]: Stopped Nix worker-protocol over gRPC.5751worker2 # [ 92.033698] systemd[1]: nix-grpc-daemon.service: Consumed 1.864s CPU time over 1min 15.520s wall clock time, 60.5M memory peak, 78.6K incoming IP traffic, 187.4K outgoing IP traffic.5752worker2 # [ 92.045313] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5753worker2 # [ 92.049681] systemd[1]: Closed Nix worker-protocol over gRPC.5754worker2 # [ 92.053225] systemd[1]: nix-grpc-daemon.socket: Consumed 319us CPU time over 1min 23.436s wall clock time, 256K memory peak, 85.5K incoming IP traffic, 73.5K outgoing IP traffic.5755lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5756lb: must succeed: curl -sf localhost:9901/clusters5757lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5758lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.05 seconds)5759client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25760worker1 # [ 92.263284] nix-grpc-daemon[780]: ts=2026-09-23T05:13:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:335905761client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5762worker1 # [ 92.268852] nix-grpc-daemon[780]: ts=2026-09-23T05:13:22Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:335905763client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.16 seconds)5764worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5765worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.02 seconds)5766worker2 # [ 92.332842] systemd[1]: Listening on Nix worker-protocol over gRPC.5767worker1 # [ 92.395428] systemd[1]: Stopping Nix worker-protocol over gRPC...5768worker2 # [ 93.023707] systemd[1]: Starting scheduler-token.service...5769worker1 # [ 93.058480] nix-grpc-daemon[780]: ts=2026-09-23T05:13:23Z level=info event=signal_exit5770worker2 # [ 93.083268] systemd[1]: scheduler-token.service: Deactivated successfully.5771worker2 # [ 93.086334] systemd[1]: Finished scheduler-token.service.5772worker2 # [ 93.089444] systemd[1]: scheduler-token.service: Consumed 34ms CPU time over 62ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5773worker2 # [ 93.097161] systemd[1]: Starting Nix worker-protocol over gRPC...5774worker2 # [ 93.175371] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:23Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5775worker2 # [ 93.177485] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:23Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5776worker2 # [ 93.180762] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:23Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15777worker2 # [ 93.185056] nix-grpc-daemon[1166]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5778worker2 # [ 93.186705] nix-grpc-daemon[1166]: E0000 00:00:1790140403.350197 1166 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5779worker2 # [ 93.192415] nix-grpc-daemon[1166]: E0000 00:00:1790140403.355627 1166 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5780worker2 # [ 93.196115] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:23Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5781worker2 # [ 93.197711] systemd[1]: Started Nix worker-protocol over gRPC.5782worker2 # [ 93.203155] nix-grpc-daemon[1166]: E0000 00:00:1790140403.367886 1209 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5783worker2 # [ 93.218064] nix-grpc-daemon[1166]: E0000 00:00:1790140403.382829 1209 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5784worker2 # [ 93.221142] nix-grpc-daemon[1166]: E0000 00:00:1790140403.382907 1209 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5785worker2 # [ 93.231584] nix-grpc-daemon[1166]: E0000 00:00:1790140403.396895 1209 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5786worker2 # [ 93.241185] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:23Z level=info event=scheduler_connected addr=lb:500515787worker2 # [ 93.242727] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5788worker2 # [ 93.444391] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:23Z level=info event=scheduler_connected addr=lb:500515789worker2 # [ 93.448295] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5790worker2 # [ 93.649397] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:23Z level=info event=scheduler_connected addr=lb:500515791worker2 # [ 93.653328] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5792lb # [ 93.713714] niks3-server[977]: 2026/09/23 05:13:23 INFO lead: released remote=192.168.1.3:570625793worker2 # [ 93.854445] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:24Z level=info event=scheduler_connected addr=lb:500515794worker2 # [ 93.858448] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5795worker2 # [ 94.059422] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:24Z level=info event=scheduler_connected addr=lb:500515796worker2 # [ 94.063338] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5797worker2 # [ 94.265159] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:24Z level=info event=scheduler_connected addr=lb:500515798worker2 # [ 94.268884] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5799worker2 # [ 94.469997] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:24Z level=info event=scheduler_connected addr=lb:500515800worker2 # [ 94.474108] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5801worker2 # [ 94.675336] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:24Z level=info event=scheduler_connected addr=lb:500515802worker2 # [ 94.679386] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5803worker1 # [ 94.721285] nix-grpc-daemon[780]: ts=2026-09-23T05:13:24Z level=info event=scheduler_restarting5804worker1 # [ 94.724977] nix-grpc-daemon[780]: ts=2026-09-23T05:13:24Z level=info event=scheduler_restarting addr=lb:500515805worker1 # [ 94.728507] nix-grpc-daemon[780]: ts=2026-09-23T05:13:24Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:566485806worker1 # [ 94.732157] nix-grpc-daemon[780]: ts=2026-09-23T05:13:24Z level=info event=worker_gone addr=192.168.1.3:500515807worker1 # [ 94.751247] nix-daemon[864]: reaped child process 866, status = succeeded5808lb # [ 94.878801] envoy[903]: conn peer=[2001:db8:1::1]:43676 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3744 tx=2098 ms=21965809lb # [ 94.883395] envoy[903]: conn peer=[2001:db8:1::1]:43688 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=421 tx=328 ms=165810lb # [ 94.886824] envoy[903]: conn peer=[2001:db8:1::1]:43708 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=460 tx=307 ms=135811lb # [ 94.890381] envoy[903]: conn peer=[2001:db8:1::1]:43704 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3741 tx=2083 ms=20265812worker2 # [ 94.880498] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:25Z level=info event=scheduler_connected addr=lb:500515813lb # [ 94.895228] envoy[903]: conn peer=[2001:db8:1::4]:39874 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=57057 tx=3058 ms=740105814lb # [ 94.898908] envoy[903]: conn peer=[2001:db8:1::1]:43716 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=519 ms=165815worker2 # [ 94.891382] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5816worker1 # [ 94.931485] nix-grpc-daemon[780]: ts=2026-09-23T05:13:24Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5817worker2 # [ 95.092688] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:25Z level=info event=scheduler_connected addr=lb:500515818worker2 # [ 95.096743] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5819worker1 # [ 95.209760] nix-grpc-daemon[780]: E0000 00:00:1790140405.202975 826 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true5820worker1 # [ 95.214723] nix-grpc-daemon[780]: E0000 00:00:1790140405.203256 826 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5821worker2 # [ 95.298702] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:25Z level=info event=scheduler_connected addr=lb:500515822worker2 # [ 95.302637] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5823worker2 # [ 95.503956] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:25Z level=info event=scheduler_connected addr=lb:500515824worker2 # [ 95.508203] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5825worker2 # [ 95.709182] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:25Z level=info event=scheduler_connected addr=lb:500515826worker2 # [ 95.713196] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5827worker2 # [ 95.914562] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:26Z level=info event=scheduler_connected addr=lb:500515828worker2 # [ 95.918684] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5829worker1 # [ 96.011345] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5830worker1 # [ 96.015440] systemd[1]: Stopped Nix worker-protocol over gRPC.5831worker1 # [ 96.019632] systemd[1]: nix-grpc-daemon.service: Consumed 2.689s CPU time over 1min 21.137s wall clock time, 69.2M memory peak, 116.8K incoming IP traffic, 252K outgoing IP traffic.5832worker1 # [ 96.031628] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5833worker1 # [ 96.035962] systemd[1]: Closed Nix worker-protocol over gRPC.5834worker1 # [ 96.040166] systemd[1]: nix-grpc-daemon.socket: Consumed 339us CPU time over 1min 28.365s wall clock time, 256K memory peak, 125.8K incoming IP traffic, 107.4K outgoing IP traffic.5835lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5836lb: must succeed: curl -sf localhost:9901/clusters5837lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5838lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.05 seconds)5839client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25840worker2 # [ 96.119973] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:26Z level=info event=scheduler_connected addr=lb:500515841worker2 # [ 96.124238] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5842worker2 # [ 96.233805] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:350565843worker2 # [ 96.242714] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)5844worker2 # [ 96.262198] nix-grpc-daemon[1166]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 86 ms (attempt 1/5)5845worker2 # [ 96.326069] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:26Z level=info event=scheduler_connected addr=lb:500515846worker2 # [ 96.330164] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5847worker2 # [ 96.355902] nix-grpc-daemon[1166]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 2 ms (attempt 2/5)5848worker2 # [ 96.364812] nix-grpc-daemon[1166]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 221 ms (attempt 3/5)5849worker2 # [ 96.531531] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:26Z level=info event=scheduler_connected addr=lb:500515850worker2 # [ 96.535623] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5851worker2 # [ 96.593881] nix-grpc-daemon[1166]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 140 ms (attempt 4/5)5852worker2 # [ 96.737247] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:26Z level=info event=scheduler_connected addr=lb:500515853worker2 # [ 96.741941] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5854worker2 # [ 96.746379] nix-grpc-daemon[1166]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5855worker2 # [ 96.749228] nix-grpc-daemon[1166]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5856client # copying 2 paths...5857client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5858worker2 # [ 96.772095] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:350565859client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5860worker2 # [ 96.782479] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)5861worker2 # [ 96.808421] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:26Z level=info event=niks3_push_started pid=12275862worker2 # [ 96.919971] nix-grpc-daemon[1227]: time=2026-09-23T05:13:27.084Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""5863worker2 # [ 96.942645] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:27Z level=info event=scheduler_connected addr=lb:500515864worker2 # [ 96.944797] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5865worker2 # [ 97.021782] nix-daemon[851]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)5866worker2 # [ 97.036489] nix-daemon[851]: reaped child process 1243, status = succeeded5867lb # [ 97.058939] niks3-server[977]: 2026/09/23 05:13:27 INFO Received uploads request method=POST path=/api/pending_closures5868lb # [ 97.070761] niks3-server[977]: 2026/09/23 05:13:27 INFO Received uploads request method=POST path=/api/pending_closures5869worker2 # [ 97.067310] nix-grpc-daemon[1227]: time=2026-09-23T05:13:27.232Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5870worker2 # [ 97.068856] nix-grpc-daemon[1227]: time=2026-09-23T05:13:27.234Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5871worker2 # [ 97.070787] nix-grpc-daemon[1227]: time=2026-09-23T05:13:27.236Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5872lb # [ 97.126149] niks3-server[977]: 2026/09/23 05:13:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5873lb # [ 97.128342] niks3-server[977]: 2026/09/23 05:13:27 INFO Signed narinfos id=35 count=15874lb # [ 97.136236] niks3-server[977]: 2026/09/23 05:13:27 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5875lb # [ 97.139401] niks3-server[977]: 2026/09/23 05:13:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5876lb # [ 97.141275] niks3-server[977]: 2026/09/23 05:13:27 INFO Signed narinfos id=36 count=25877worker2 # [ 97.129605] nix-grpc-daemon[1227]: time=2026-09-23T05:13:27.294Z level=INFO msg="Uploading 3 narinfos"5878lb # [ 97.142856] niks3-server[977]: 2026/09/23 05:13:27 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5879lb # [ 97.147496] niks3-server[977]: 2026/09/23 05:13:27 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5880lb # [ 97.150576] niks3-server[977]: 2026/09/23 05:13:27 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5881worker2 # [ 97.145659] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:27Z level=info event=scheduler_connected addr=lb:500515882lb # [ 97.161807] niks3-server[977]: 2026/09/23 05:13:27 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5883worker2 # [ 97.149270] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5884lb # [ 97.164975] niks3-server[977]: 2026/09/23 05:13:27 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5885lb # [ 97.168556] niks3-server[977]: 2026/09/23 05:13:27 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5886lb # [ 97.171188] niks3-server[977]: 2026/09/23 05:13:27 INFO Completed upload id=355887lb # [ 97.172420] niks3-server[977]: 2026/09/23 05:13:27 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5888lb # [ 97.175661] niks3-server[977]: 2026/09/23 05:13:27 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5889lb # [ 97.178158] niks3-server[977]: 2026/09/23 05:13:27 INFO Completed upload id=365890worker2 # [ 97.166918] nix-grpc-daemon[1227]: time=2026-09-23T05:13:27.332Z level=INFO msg="Upload complete. (243ms)"5891worker2 # [ 97.171686] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35056 duration_s=0 paths=2 nar_bytes_in=3125892worker2 # [ 97.184187] nix-daemon[851]: reaped child process 1225, status = succeeded5893client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.12 seconds)5894worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5895worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.02 seconds)5896worker1 # [ 97.283093] systemd[1]: Listening on Nix worker-protocol over gRPC.5897worker1 # [ 97.291600] systemd[1]: Starting Nix worker-protocol over gRPC...5898worker2 # [ 97.352304] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:27Z level=info event=scheduler_connected addr=lb:500515899worker2 # [ 97.355524] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5900worker1 # [ 97.378401] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:27Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5901worker1 # [ 97.380686] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:27Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5902worker1 # [ 97.383864] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:27Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15903worker1 # [ 97.389056] nix-grpc-daemon[1334]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5904worker1 # [ 97.390415] nix-grpc-daemon[1334]: E0000 00:00:1790140407.381636 1334 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5905worker1 # [ 97.396647] nix-grpc-daemon[1334]: E0000 00:00:1790140407.387113 1334 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5906worker1 # [ 97.400744] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:27Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5907worker1 # [ 97.403603] systemd[1]: Started Nix worker-protocol over gRPC.5908lb # [ 97.402590] niks3-server[977]: 2026/09/23 05:13:27 INFO lead: acquired remote=192.168.1.3:397445909worker1 # [ 97.408139] nix-grpc-daemon[1334]: E0000 00:00:1790140407.399089 1377 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5910worker1 # [ 97.413085] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:27Z level=info event=scheduler_take_over5911worker1 # [ 97.419876] nix-grpc-daemon[1334]: E0000 00:00:1790140407.413423 1377 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5912worker1 # [ 97.424055] nix-grpc-daemon[1334]: E0000 00:00:1790140407.415764 1377 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5913lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5914lb: must succeed: curl -sf localhost:9901/clusters5915worker1 # [ 97.435243] nix-grpc-daemon[1334]: E0000 00:00:1790140407.428778 1377 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5916worker1 # [ 97.446387] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:27Z level=info event=scheduler_connected addr=lb:500515917worker1 # [ 97.448078] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5918lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5919worker2 # [ 97.553749] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:27Z level=info event=scheduler_connected addr=lb:500515920worker2 # [ 97.557742] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5921worker1 # [ 97.649429] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:27Z level=info event=scheduler_connected addr=lb:500515922worker1 # [ 97.653580] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5923worker2 # [ 97.759187] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:27Z level=info event=scheduler_connected addr=lb:500515924worker2 # [ 97.763281] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5925worker1 # [ 97.854573] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:27Z level=info event=scheduler_connected addr=lb:500515926worker1 # [ 97.858834] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5927worker2 # [ 97.964643] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:28Z level=info event=scheduler_connected addr=lb:500515928worker2 # [ 97.968741] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5929worker1 # [ 98.059536] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:28Z level=info event=scheduler_connected addr=lb:500515930worker1 # [ 98.066150] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:28Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:607885931worker1 # [ 98.070658] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:28Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05932worker2 # [ 98.170201] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:28Z level=info event=scheduler_connected addr=lb:500515933worker1 # [ 98.193185] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:28Z level=info event=oidc_jwks provider=mock keys=15934worker1 # [ 98.197349] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:28Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:607885935worker1 # [ 98.201388] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:28Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05936lb: must succeed: curl -sf localhost:9901/clusters5937lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5938lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.10 seconds)5939(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 10.52 seconds)5940subtest: a niks3 restart does not cost the leader its role5941lb: must succeed: curl -sf localhost:9901/clusters5942lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5943lb: must succeed: curl -sf localhost:9901/clusters5944lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5945lb # [ 98.599327] niks3-server[977]: 2026/09/23 05:13:28 INFO Shutdown signal received, draining in-flight requests timeout=10s5946lb # [ 98.601039] niks3-server[977]: 2026/09/23 05:13:28 INFO lead: released remote=192.168.1.3:397445947lb # [ 98.602262] systemd[1]: Stopping niks3 server...5948worker1 # [ 98.602654] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:28Z level=info event=scheduler_lock_lost status=2005949worker2 # [ 98.589477] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:28Z level=info event=scheduler_lock_lost status=2005950lb # [ 98.613801] systemd[1]: niks3.service: Deactivated successfully.5951lb # [ 98.615337] systemd[1]: Stopped niks3 server.5952lb # [ 98.616176] systemd[1]: niks3.service: Consumed 791ms CPU time over 1min 19.153s wall clock time, 28.8M memory peak, 100K incoming IP traffic, 142.9K outgoing IP traffic.5953lb # [ 98.623288] systemd[1]: Starting niks3 server...5954lb # [ 98.672623] niks3-server[1118]: 2026/09/23 05:13:28 goose: no migrations to run. current version: 202609200000005955lb # [ 98.678191] niks3-server[1118]: 2026/09/23 05:13:28 OK 1_commit_pending_closure.sql (3.95ms)5956lb # [ 98.681426] niks3-server[1118]: 2026/09/23 05:13:28 OK 2_object_stats_trigger.sql (1.97ms)5957lb # [ 98.682590] niks3-server[1118]: 2026/09/23 05:13:28 goose: up to current file version: 25958lb # [ 98.684711] niks3-server[1118]: 2026/09/23 05:13:28 INFO Loaded signing key name=farm-test-1 path=/nix/store/hk0nhypl37jvkn3gr36v3y7qx593p39i-key5959lb # [ 98.695440] niks3-server[1118]: 2026/09/23 05:13:28 INFO Read proxy enabled — serving cache objects from S35960lb # [ 98.696750] niks3-server[1118]: 2026/09/23 05:13:28 INFO Using socket-activated listener address=0.0.0.0:57515961lb # [ 98.698437] systemd[1]: Started niks3 server.5962lb # [ 98.699863] niks3-server[1118]: 2026/09/23 05:13:28 INFO systemd watchdog enabled interval=15s5963lb # [ 98.700966] niks3-server[1118]: 2026/09/23 05:13:28 INFO Starting HTTP server address=0.0.0.0:57515964lb: waiting for unit niks3.service5965lb: (finished: waiting for unit niks3.service, in 0.02 seconds)5966lb: must succeed: curl -sf localhost:9901/clusters5967lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5968worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5969worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5970lb: must succeed: curl -sf localhost:9901/clusters5971lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5972lb: must succeed: curl -sf localhost:9901/clusters5973lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5974worker1: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5975worker1: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.05 seconds)5976client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&25977worker2 # [ 99.015314] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:29Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:350565978client # don't know how to build these paths:5979client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5980worker2 # [ 99.024901] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:350565981worker1 # [ 99.064312] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:29Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=05982worker1 # [ 99.070132] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:29Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.3:50051 clients=15983worker1 # [ 99.113739] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608045984worker1 # [ 99.124969] nix-daemon[864]: accepted connection from pid 1334, user nix-grpc-daemon (trusted)5985worker1 # [ 99.141407] nix-grpc-daemon[1334]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 28 ms (attempt 1/5)5986worker1 # [ 99.176774] nix-grpc-daemon[1334]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 9 ms (attempt 2/5)5987worker1 # [ 99.187400] nix-grpc-daemon[1334]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 249 ms (attempt 3/5)5988worker1 # [ 99.444077] nix-grpc-daemon[1334]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 244 ms (attempt 4/5)5989lb # [ 99.606121] niks3-server[1118]: 2026/09/23 05:13:29 INFO lead: acquired remote=192.168.1.3:397625990worker1 # [ 99.692644] nix-grpc-daemon[1334]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5991worker1 # [ 99.696498] nix-grpc-daemon[1334]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5992worker1 # [ 99.713323] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608045993client # copying 1 paths to 'lb:50051'...5994client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...5995worker1 # [ 99.728401] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608045996worker1 # [ 99.738179] nix-daemon[864]: accepted connection from pid 1334, user nix-grpc-daemon (trusted)5997worker1 # [ 99.760483] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:29Z level=info event=niks3_push_started pid=14075998worker1 # [ 99.863481] nix-grpc-daemon[1407]: time=2026-09-23T05:13:29.856Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""5999worker1 # [ 99.959882] nix-daemon[864]: accepted connection from pid 1420, user nix-grpc-daemon (trusted)6000worker1 # [ 99.973879] nix-daemon[864]: reaped child process 1423, status = succeeded6001lb # [ 99.979732] niks3-server[1118]: 2026/09/23 05:13:30 INFO Received uploads request method=POST path=/api/pending_closures6002worker1 # [ 99.993898] nix-grpc-daemon[1407]: time=2026-09-23T05:13:29.987Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6003worker1 # [ 99.995446] nix-grpc-daemon[1407]: time=2026-09-23T05:13:29.989Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"6004lb # [ 100.029225] niks3-server[1118]: 2026/09/23 05:13:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6005lb # [ 100.035778] niks3-server[1118]: 2026/09/23 05:13:30 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst6006lb # [ 100.054815] niks3-server[1118]: 2026/09/23 05:13:30 INFO Signed narinfos id=37 count=16007worker1 # [ 100.058934] nix-grpc-daemon[1407]: time=2026-09-23T05:13:30.052Z level=INFO msg="Uploading 1 narinfos"6008lb # [ 100.061497] niks3-server[1118]: 2026/09/23 05:13:30 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls6009lb # [ 100.067107] niks3-server[1118]: 2026/09/23 05:13:30 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6010lb # [ 100.073373] niks3-server[1118]: 2026/09/23 05:13:30 INFO Completed upload id=376011worker1 # [ 100.076047] nix-grpc-daemon[1407]: time=2026-09-23T05:13:30.069Z level=INFO msg="Upload complete. (209ms)"6012lb # [ 100.078091] niks3-server[1118]: 2026/09/23 05:13:30 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo6013worker1 # [ 100.081058] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60804 duration_s=0 paths=1 nar_bytes_in=5366014worker1 # [ 100.087518] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608046015worker1 # [ 100.099114] nix-daemon[864]: accepted connection from pid 1334, user nix-grpc-daemon (trusted)6016worker1 # [ 100.106531] nix-daemon[864]: reaped child process 1405, status = succeeded6017worker1 # [ 100.111176] nix-daemon[864]: accepted connection from pid 1334, user nix-grpc-daemon (trusted)6018client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...6019client # farm-a> node-a: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv6020client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6021client # farm-a> LOG-farm-a-after-niks3-restart6022worker1 # [ 101.347451] nix-daemon[864]: reaped child process 1427, status = succeeded6023worker1 # [ 101.432197] nix-daemon[864]: accepted connection from pid 1443, user nix-grpc-daemon (trusted)6024worker1 # [ 101.445930] nix-daemon[864]: reaped child process 1446, status = succeeded6025lb # [ 101.449462] niks3-server[1118]: 2026/09/23 05:13:31 INFO Received uploads request method=POST path=/api/pending_closures6026worker1 # [ 101.467329] nix-grpc-daemon[1407]: time=2026-09-23T05:13:31.459Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6027worker1 # [ 101.468754] nix-grpc-daemon[1407]: time=2026-09-23T05:13:31.459Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"6028lb # [ 101.487481] niks3-server[1118]: 2026/09/23 05:13:31 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv6029lb # [ 101.503842] niks3-server[1118]: 2026/09/23 05:13:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6030lb # [ 101.507680] niks3-server[1118]: 2026/09/23 05:13:31 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst6031worker1 # [ 101.517846] nix-grpc-daemon[1407]: time=2026-09-23T05:13:31.511Z level=INFO msg="Uploading 1 narinfos"6032lb # [ 101.515111] niks3-server[1118]: 2026/09/23 05:13:31 INFO Signed narinfos id=38 count=16033lb # [ 101.523465] niks3-server[1118]: 2026/09/23 05:13:31 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls6034lb # [ 101.528245] niks3-server[1118]: 2026/09/23 05:13:31 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6035lb # [ 101.530810] niks3-server[1118]: 2026/09/23 05:13:31 INFO Completed upload id=386036worker1 # [ 101.534650] nix-grpc-daemon[1407]: time=2026-09-23T05:13:31.528Z level=INFO msg="Upload complete. (215ms)"6037lb # [ 101.534137] niks3-server[1118]: 2026/09/23 05:13:31 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo6038worker1 # [ 101.541624] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60804 duration_s=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=16039worker1 # [ 101.553392] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:31Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=06040worker1 # [ 101.556970] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:31Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.3:50051 clients=16041worker1 # [ 101.565954] nix-daemon[864]: reaped child process 1425, status = succeeded6042worker1 # [ 101.598292] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608046043worker1 # [ 101.610749] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608046044worker1 # [ 101.623955] nix-grpc-daemon[1407]: time=2026-09-23T05:13:31.617Z level=INFO msg="All 1 paths already cached"6045client # copying 1 paths to 'lb:50051'...6046client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...6047worker1 # [ 101.632788] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608046048worker1 # [ 101.641170] nix-daemon[864]: accepted connection from pid 1334, user nix-grpc-daemon (trusted)6049worker1 # [ 101.737511] nix-daemon[864]: accepted connection from pid 1452, user nix-grpc-daemon (trusted)6050worker1 # [ 101.751430] nix-daemon[864]: reaped child process 1455, status = succeeded6051lb # [ 101.755412] niks3-server[1118]: 2026/09/23 05:13:31 INFO Received uploads request method=POST path=/api/pending_closures6052worker1 # [ 101.767795] nix-grpc-daemon[1407]: time=2026-09-23T05:13:31.761Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6053worker1 # [ 101.769584] nix-grpc-daemon[1407]: time=2026-09-23T05:13:31.762Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"6054lb # [ 101.786837] niks3-server[1118]: 2026/09/23 05:13:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6055lb # [ 101.795250] niks3-server[1118]: 2026/09/23 05:13:31 INFO Signed narinfos id=39 count=16056worker1 # [ 101.800787] nix-grpc-daemon[1407]: time=2026-09-23T05:13:31.794Z level=INFO msg="Uploading 1 narinfos"6057lb # [ 101.803778] niks3-server[1118]: 2026/09/23 05:13:31 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls6058lb # [ 101.814622] niks3-server[1118]: 2026/09/23 05:13:31 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst6059lb # [ 101.821756] niks3-server[1118]: 2026/09/23 05:13:31 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6060lb # [ 101.829960] niks3-server[1118]: 2026/09/23 05:13:31 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo6061lb # [ 101.833700] niks3-server[1118]: 2026/09/23 05:13:31 INFO Completed upload id=396062worker1 # [ 101.836352] nix-grpc-daemon[1407]: time=2026-09-23T05:13:31.829Z level=INFO msg="Upload complete. (182ms)"6063worker1 # [ 101.840978] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60804 duration_s=0 paths=1 nar_bytes_in=6566064worker1 # [ 101.848115] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608046065worker1 # [ 101.858335] nix-daemon[864]: accepted connection from pid 1334, user nix-grpc-daemon (trusted)6066worker1 # [ 101.868378] nix-daemon[864]: reaped child process 1450, status = succeeded6067worker1 # [ 101.874193] nix-daemon[864]: accepted connection from pid 1334, user nix-grpc-daemon (trusted)6068client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...6069client # farm-b> node-a: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv6070client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6071client # farm-b> LOG-farm-b-after-niks3-restart6072worker1 # [ 102.709776] nix-daemon[864]: reaped child process 1459, status = succeeded6073worker1 # [ 102.794946] nix-daemon[864]: accepted connection from pid 1475, user nix-grpc-daemon (trusted)6074worker1 # [ 102.807067] nix-daemon[864]: reaped child process 1478, status = succeeded6075lb # [ 102.812382] niks3-server[1118]: 2026/09/23 05:13:32 INFO Received uploads request method=POST path=/api/pending_closures6076worker1 # [ 102.823744] nix-grpc-daemon[1407]: time=2026-09-23T05:13:32.817Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6077worker1 # [ 102.825303] nix-grpc-daemon[1407]: time=2026-09-23T05:13:32.818Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"6078lb # [ 102.852639] niks3-server[1118]: 2026/09/23 05:13:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6079lb # [ 102.861201] niks3-server[1118]: 2026/09/23 05:13:32 INFO Signed narinfos id=40 count=16080worker1 # [ 102.870916] nix-grpc-daemon[1407]: time=2026-09-23T05:13:32.864Z level=INFO msg="Uploading 1 narinfos"6081lb # [ 102.872076] niks3-server[1118]: 2026/09/23 05:13:32 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst6082lb # [ 102.880507] niks3-server[1118]: 2026/09/23 05:13:32 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls6083lb # [ 102.887149] niks3-server[1118]: 2026/09/23 05:13:32 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv6084lb # [ 102.893582] niks3-server[1118]: 2026/09/23 05:13:32 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6085lb # [ 102.898457] niks3-server[1118]: 2026/09/23 05:13:32 INFO Completed upload id=406086lb # [ 102.900662] niks3-server[1118]: 2026/09/23 05:13:32 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo6087worker1 # [ 102.902565] nix-grpc-daemon[1407]: time=2026-09-23T05:13:32.896Z level=INFO msg="Upload complete. (222ms)"6088worker1 # [ 102.911570] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60804 duration_s=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=16089worker1 # [ 102.921816] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:32Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=06090worker1 # [ 102.927196] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:32Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.3:50051 clients=16091worker1 # [ 102.936500] nix-daemon[864]: reaped child process 1457, status = succeeded6092worker1 # [ 102.967279] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608046093worker1 # [ 102.979913] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608046094worker1 # [ 102.993560] nix-grpc-daemon[1407]: time=2026-09-23T05:13:32.987Z level=INFO msg="All 2 paths already cached"6095client # copying 1 paths to 'lb:50051'...6096client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...6097worker1 # [ 103.002854] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608046098worker1 # [ 103.011285] nix-daemon[864]: accepted connection from pid 1334, user nix-grpc-daemon (trusted)6099worker1 # [ 103.109141] nix-daemon[864]: accepted connection from pid 1484, user nix-grpc-daemon (trusted)6100worker1 # [ 103.122965] nix-daemon[864]: reaped child process 1487, status = succeeded6101lb # [ 103.127634] niks3-server[1118]: 2026/09/23 05:13:33 INFO Received uploads request method=POST path=/api/pending_closures6102worker1 # [ 103.138257] nix-grpc-daemon[1407]: time=2026-09-23T05:13:33.131Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6103worker1 # [ 103.139784] nix-grpc-daemon[1407]: time=2026-09-23T05:13:33.133Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"6104lb # [ 103.162234] niks3-server[1118]: 2026/09/23 05:13:33 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls6105lb # [ 103.166560] niks3-server[1118]: 2026/09/23 05:13:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6106lb # [ 103.172369] niks3-server[1118]: 2026/09/23 05:13:33 INFO Signed narinfos id=41 count=16107worker1 # [ 103.178184] nix-grpc-daemon[1407]: time=2026-09-23T05:13:33.171Z level=INFO msg="Uploading 1 narinfos"6108lb # [ 103.181434] niks3-server[1118]: 2026/09/23 05:13:33 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst6109lb # [ 103.193736] niks3-server[1118]: 2026/09/23 05:13:33 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6110lb # [ 103.202429] niks3-server[1118]: 2026/09/23 05:13:33 INFO Completed upload id=416111worker1 # [ 103.204680] nix-grpc-daemon[1407]: time=2026-09-23T05:13:33.198Z level=INFO msg="Upload complete. (179ms)"6112lb # [ 103.205897] niks3-server[1118]: 2026/09/23 05:13:33 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo6113worker1 # [ 103.209531] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60804 duration_s=0 paths=1 nar_bytes_in=7846114worker1 # [ 103.215151] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608046115worker1 # [ 103.228224] nix-daemon[864]: accepted connection from pid 1334, user nix-grpc-daemon (trusted)6116worker1 # [ 103.231780] nix-daemon[864]: reaped child process 1482, status = succeeded6117worker1 # [ 103.237120] nix-daemon[864]: accepted connection from pid 1334, user nix-grpc-daemon (trusted)6118client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...6119client # farm-top> node-a: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6120client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6121client # farm-top> LOG-farm-top-after-niks3-restart6122worker1 # [ 104.476804] nix-daemon[864]: reaped child process 1491, status = succeeded6123worker1 # [ 104.560899] nix-daemon[864]: accepted connection from pid 1507, user nix-grpc-daemon (trusted)6124worker1 # [ 104.574942] nix-daemon[864]: reaped child process 1510, status = succeeded6125lb # [ 104.579695] niks3-server[1118]: 2026/09/23 05:13:34 INFO Received uploads request method=POST path=/api/pending_closures6126worker1 # [ 104.593144] nix-grpc-daemon[1407]: time=2026-09-23T05:13:34.586Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6127worker1 # [ 104.594654] nix-grpc-daemon[1407]: time=2026-09-23T05:13:34.588Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"6128lb # [ 104.620770] niks3-server[1118]: 2026/09/23 05:13:34 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6129lb # [ 104.629988] niks3-server[1118]: 2026/09/23 05:13:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6130worker1 # [ 104.641854] nix-grpc-daemon[1407]: time=2026-09-23T05:13:34.635Z level=INFO msg="Uploading 1 narinfos"6131lb # [ 104.639197] niks3-server[1118]: 2026/09/23 05:13:34 INFO Signed narinfos id=42 count=16132lb # [ 104.644257] niks3-server[1118]: 2026/09/23 05:13:34 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls6133lb # [ 104.648433] niks3-server[1118]: 2026/09/23 05:13:34 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst6134lb # [ 104.658590] niks3-server[1118]: 2026/09/23 05:13:34 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6135lb # [ 104.662583] niks3-server[1118]: 2026/09/23 05:13:34 INFO Completed upload id=426136worker1 # [ 104.665286] nix-grpc-daemon[1407]: time=2026-09-23T05:13:34.658Z level=INFO msg="Upload complete. (217ms)"6137lb # [ 104.665700] niks3-server[1118]: 2026/09/23 05:13:34 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo6138worker1 # [ 104.674103] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60804 duration_s=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=16139worker1 # [ 104.688082] nix-daemon[864]: reaped child process 1489, status = succeeded6140client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&2, in 5.82 seconds)6141client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath6142client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath, in 0.12 seconds)6143(finished: subtest: a niks3 restart does not cost the leader its role, in 6.29 seconds)6144subtest: scheduler down: the other node takes the lock and keeps it6145lb: must succeed: curl -sf localhost:9901/clusters6146lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6147lb: must succeed: curl -sf localhost:9901/clusters6148lb # [ 104.878253] envoy[903]: conn peer=[2001:db8:1::3]:47750 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=17298 tx=3618 ms=798406149lb # [ 104.882244] envoy[903]: conn peer=[2001:db8:1::1]:45282 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1008 tx=282 ms=9516150lb # [ 104.886515] envoy[903]: conn peer=[2001:db8:1::1]:45298 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9696 tx=4412 ms=56576151lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6152lb: must succeed: curl -sf localhost:9901/clusters6153lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6154lb: must succeed: curl -sf localhost:9901/clusters6155lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6156worker1 # [ 104.994603] systemd[1]: Stopping Nix worker-protocol over gRPC...6157worker1 # [ 105.398319] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:35Z level=info event=signal_exit6158lb # [ 105.610841] niks3-server[1118]: 2026/09/23 05:13:35 INFO lead: released remote=192.168.1.3:397626159worker2 # [ 106.599546] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:36Z level=info event=scheduler_restarting addr=lb:500516160worker1 # [ 106.616930] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:36Z level=info event=scheduler_restarting6161worker1 # [ 106.621624] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:36Z level=info event=scheduler_restarting addr=lb:500516162worker1 # [ 106.624805] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:36Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:607886163worker1 # [ 106.628529] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:36Z level=info event=worker_gone addr=192.168.1.3:500516164worker1 # [ 106.631813] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:36Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:607886165worker1 # [ 106.635598] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:36Z level=info event=worker_gone addr=192.168.1.4:500516166worker1 # [ 106.652372] nix-daemon[864]: reaped child process 1397, status = succeeded6167worker2 # [ 106.800099] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:36Z level=info event=scheduler_connected addr=lb:500516168worker2 # [ 106.804141] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6169worker1 # [ 106.825393] nix-grpc-daemon[1334]: ts=2026-09-23T05:13:36Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6170worker2 # [ 107.006206] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:37Z level=info event=scheduler_connected addr=lb:500516171worker2 # [ 107.011139] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6172worker1 # [ 107.035937] nix-grpc-daemon[1334]: E0000 00:00:1790140417.029130 1380 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 10, started true, shutdown true6173worker1 # [ 107.041139] nix-grpc-daemon[1334]: E0000 00:00:1790140417.029411 1380 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6174worker2 # [ 107.212216] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:37Z level=info event=scheduler_connected addr=lb:500516175worker2 # [ 107.216271] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6176worker2 # [ 107.417667] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:37Z level=info event=scheduler_connected addr=lb:500516177worker2 # [ 107.421771] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6178worker2 # [ 107.623851] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:37Z level=info event=scheduler_connected addr=lb:500516179worker2 # [ 107.627726] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6180worker2 # [ 107.826192] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:37Z level=info event=scheduler_connected addr=lb:500516181worker2 # [ 107.830294] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6182worker2 # [ 108.031608] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:38Z level=info event=scheduler_connected addr=lb:500516183worker2 # [ 108.035677] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6184worker2 # [ 108.236910] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:38Z level=info event=scheduler_connected addr=lb:500516185worker2 # [ 108.241269] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6186worker2 # [ 108.442174] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:38Z level=info event=scheduler_connected addr=lb:500516187worker2 # [ 108.445537] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6188worker2 # [ 108.647282] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:38Z level=info event=scheduler_connected addr=lb:500516189worker2 # [ 108.650507] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6190worker1 # [ 108.827369] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6191worker1 # [ 108.831360] systemd[1]: Stopped Nix worker-protocol over gRPC.6192worker1 # [ 108.835436] systemd[1]: nix-grpc-daemon.service: Consumed 830ms CPU time over 11.539s wall clock time, 36.6M memory peak, 36.1K incoming IP traffic, 60.8K outgoing IP traffic.6193worker1 # [ 108.847094] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.6194worker1 # [ 108.851486] systemd[1]: Closed Nix worker-protocol over gRPC.6195worker1 # [ 108.854903] systemd[1]: nix-grpc-daemon.socket: Consumed 675us CPU time over 11.575s wall clock time, 256K memory peak, 32.8K incoming IP traffic, 23.8K outgoing IP traffic.6196worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over6197worker2 # [ 108.851620] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:39Z level=info event=scheduler_connected addr=lb:500516198worker2 # [ 108.855077] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6199worker2 # [ 109.056134] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:39Z level=info event=scheduler_connected addr=lb:500516200worker2 # [ 109.059472] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6201worker2 # [ 109.260461] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:39Z level=info event=scheduler_connected addr=lb:500516202worker2 # [ 109.264408] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6203worker2 # [ 109.465920] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:39Z level=info event=scheduler_connected addr=lb:500516204worker2 # [ 109.469947] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6205lb # [ 109.607830] niks3-server[1118]: 2026/09/23 05:13:39 INFO lead: acquired remote=192.168.1.4:512986206worker2 # [ 109.599715] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:39Z level=info event=scheduler_take_over6207worker2 # [ 109.672316] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:39Z level=info event=scheduler_connected addr=lb:500516208worker2 # [ 109.675975] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6209worker2 # [ 109.877393] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:40Z level=info event=scheduler_connected addr=lb:500516210worker2 # [ 109.881722] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6211worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 1.10 seconds)6212worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6213worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)6214worker2 # [ 110.082926] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:40Z level=info event=scheduler_connected addr=lb:500516215worker2 # [ 110.087708] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6216worker2 # [ 110.288202] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:40Z level=info event=scheduler_connected addr=lb:500516217worker2 # [ 110.307412] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:40Z level=info event=oidc_jwks provider=mock keys=16218worker2 # [ 110.311377] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:40Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:329086219worker2 # [ 110.315668] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:40Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06220worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6221worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6222client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&26223client # don't know how to build these paths:6224client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6225worker2 # [ 111.167526] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:350566226worker2 # [ 111.173363] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:350566227worker2 # [ 111.195708] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:41Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=06228worker2 # [ 111.201664] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:41Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.4:50051 clients=16229worker2 # [ 111.240703] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:350566230worker2 # [ 111.254200] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:350566231client # copying 1 paths to 'lb:50051'...6232client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...6233worker2 # [ 111.267134] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:350566234worker2 # [ 111.279404] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)6235worker2 # [ 111.385847] nix-daemon[851]: accepted connection from pid 1272, user nix-grpc-daemon (trusted)6236worker2 # [ 111.402327] nix-daemon[851]: reaped child process 1275, status = succeeded6237lb # [ 111.421380] niks3-server[1118]: 2026/09/23 05:13:41 INFO Received uploads request method=POST path=/api/pending_closures6238worker2 # [ 111.419202] nix-grpc-daemon[1227]: time=2026-09-23T05:13:41.583Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6239worker2 # [ 111.420595] nix-grpc-daemon[1227]: time=2026-09-23T05:13:41.583Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"6240lb # [ 111.455508] niks3-server[1118]: 2026/09/23 05:13:41 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst6241lb # [ 111.469941] niks3-server[1118]: 2026/09/23 05:13:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6242worker2 # [ 111.462835] nix-grpc-daemon[1227]: time=2026-09-23T05:13:41.627Z level=INFO msg="Uploading 1 narinfos"6243lb # [ 111.473987] niks3-server[1118]: 2026/09/23 05:13:41 INFO Signed narinfos id=43 count=16244lb # [ 111.484401] niks3-server[1118]: 2026/09/23 05:13:41 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls6245lb # [ 111.491436] niks3-server[1118]: 2026/09/23 05:13:41 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6246lb # [ 111.495512] niks3-server[1118]: 2026/09/23 05:13:41 INFO Completed upload id=436247worker2 # [ 111.484070] nix-grpc-daemon[1227]: time=2026-09-23T05:13:41.649Z level=INFO msg="Upload complete. (192ms)"6248lb # [ 111.500340] niks3-server[1118]: 2026/09/23 05:13:41 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo6249worker2 # [ 111.487966] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35056 duration_s=0 paths=1 nar_bytes_in=5206250worker2 # [ 111.493269] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:350566251worker2 # [ 111.502930] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)6252worker2 # [ 111.509363] nix-daemon[851]: reaped child process 1270, status = succeeded6253client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...6254worker2 # [ 111.514748] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)6255client # farm-a> worker2: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6256client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6257client # farm-a> LOG-farm-a-on-standby6258worker2 # [ 112.358408] nix-daemon[851]: reaped child process 1279, status = succeeded6259worker2 # [ 112.442103] nix-daemon[851]: accepted connection from pid 1295, user nix-grpc-daemon (trusted)6260worker2 # [ 112.456371] nix-daemon[851]: reaped child process 1298, status = succeeded6261lb # [ 112.474844] niks3-server[1118]: 2026/09/23 05:13:42 INFO Received uploads request method=POST path=/api/pending_closures6262worker2 # [ 112.472630] nix-grpc-daemon[1227]: time=2026-09-23T05:13:42.637Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6263worker2 # [ 112.474150] nix-grpc-daemon[1227]: time=2026-09-23T05:13:42.639Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"6264lb # [ 112.517533] niks3-server[1118]: 2026/09/23 05:13:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6265lb # [ 112.519119] niks3-server[1118]: 2026/09/23 05:13:42 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst6266worker2 # [ 112.510127] nix-grpc-daemon[1227]: time=2026-09-23T05:13:42.675Z level=INFO msg="Uploading 1 narinfos"6267lb # [ 112.523170] niks3-server[1118]: 2026/09/23 05:13:42 INFO Signed narinfos id=44 count=16268lb # [ 112.525641] niks3-server[1118]: 2026/09/23 05:13:42 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6269lb # [ 112.535972] niks3-server[1118]: 2026/09/23 05:13:42 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6270lb # [ 112.541858] niks3-server[1118]: 2026/09/23 05:13:42 INFO Completed upload id=446271worker2 # [ 112.530404] nix-grpc-daemon[1227]: time=2026-09-23T05:13:42.695Z level=INFO msg="Upload complete. (209ms)"6272lb # [ 112.544034] niks3-server[1118]: 2026/09/23 05:13:42 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo6273lb # [ 112.549691] niks3-server[1118]: 2026/09/23 05:13:42 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls6274worker2 # [ 112.536785] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35056 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=16275worker2 # [ 112.544984] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:42Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=06276worker2 # [ 112.548142] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:42Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.4:50051 clients=16277worker2 # [ 112.554141] nix-daemon[851]: reaped child process 1277, status = succeeded6278worker2 # [ 112.589100] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:350566279worker2 # [ 112.599980] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:350566280worker2 # [ 112.612614] nix-grpc-daemon[1227]: time=2026-09-23T05:13:42.777Z level=INFO msg="All 1 paths already cached"6281client # copying 1 paths to 'lb:50051'...6282client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...6283worker2 # [ 112.620727] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:350566284worker2 # [ 112.629142] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)6285worker2 # [ 112.726899] nix-daemon[851]: accepted connection from pid 1304, user nix-grpc-daemon (trusted)6286worker2 # [ 112.742095] nix-daemon[851]: reaped child process 1307, status = succeeded6287lb # [ 112.761805] niks3-server[1118]: 2026/09/23 05:13:42 INFO Received uploads request method=POST path=/api/pending_closures6288worker2 # [ 112.760391] nix-grpc-daemon[1227]: time=2026-09-23T05:13:42.925Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6289worker2 # [ 112.761887] nix-grpc-daemon[1227]: time=2026-09-23T05:13:42.927Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"6290lb # [ 112.797427] niks3-server[1118]: 2026/09/23 05:13:42 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst6291lb # [ 112.802974] niks3-server[1118]: 2026/09/23 05:13:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6292lb # [ 112.807881] niks3-server[1118]: 2026/09/23 05:13:42 INFO Signed narinfos id=45 count=16293worker2 # [ 112.799469] nix-grpc-daemon[1227]: time=2026-09-23T05:13:42.964Z level=INFO msg="Uploading 1 narinfos"6294lb # [ 112.821102] niks3-server[1118]: 2026/09/23 05:13:42 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls6295lb # [ 112.827221] niks3-server[1118]: 2026/09/23 05:13:42 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6296lb # [ 112.838806] niks3-server[1118]: 2026/09/23 05:13:42 INFO Completed upload id=456297worker2 # [ 112.827317] nix-grpc-daemon[1227]: time=2026-09-23T05:13:42.992Z level=INFO msg="Upload complete. (188ms)"6298lb # [ 112.840968] niks3-server[1118]: 2026/09/23 05:13:42 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo6299worker2 # [ 112.832357] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35056 duration_s=0 paths=1 nar_bytes_in=6406300worker2 # [ 112.838415] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:350566301worker2 # [ 112.850116] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)6302worker2 # [ 112.858247] nix-daemon[851]: reaped child process 1302, status = succeeded6303worker2 # [ 112.862937] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)6304client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...6305client # farm-b> worker2: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6306client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6307client # farm-b> LOG-farm-b-on-standby6308worker2 # [ 114.092178] nix-daemon[851]: reaped child process 1311, status = succeeded6309worker2 # [ 114.177490] nix-daemon[851]: accepted connection from pid 1327, user nix-grpc-daemon (trusted)6310worker2 # [ 114.191991] nix-daemon[851]: reaped child process 1330, status = succeeded6311lb # [ 114.213276] niks3-server[1118]: 2026/09/23 05:13:44 INFO Received uploads request method=POST path=/api/pending_closures6312worker2 # [ 114.209904] nix-grpc-daemon[1227]: time=2026-09-23T05:13:44.375Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6313worker2 # [ 114.212041] nix-grpc-daemon[1227]: time=2026-09-23T05:13:44.376Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"6314lb # [ 114.253810] niks3-server[1118]: 2026/09/23 05:13:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6315lb # [ 114.259649] niks3-server[1118]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls6316lb # [ 114.270589] niks3-server[1118]: 2026/09/23 05:13:44 INFO Signed narinfos id=46 count=16317worker2 # [ 114.261563] nix-grpc-daemon[1227]: time=2026-09-23T05:13:44.426Z level=INFO msg="Uploading 1 narinfos"6318lb # [ 114.273990] niks3-server[1118]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6319lb # [ 114.279112] niks3-server[1118]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst6320lb # [ 114.291242] niks3-server[1118]: 2026/09/23 05:13:44 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6321lb # [ 114.297733] niks3-server[1118]: 2026/09/23 05:13:44 INFO Completed upload id=466322worker2 # [ 114.285937] nix-grpc-daemon[1227]: time=2026-09-23T05:13:44.451Z level=INFO msg="Upload complete. (214ms)"6323lb # [ 114.301966] niks3-server[1118]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo6324worker2 # [ 114.292575] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35056 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=16325worker2 # [ 114.305645] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:44Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=06326worker2 # [ 114.310224] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:44Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.4:50051 clients=16327worker2 # [ 114.320344] nix-daemon[851]: reaped child process 1309, status = succeeded6328worker2 # [ 114.350595] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:350566329worker2 # [ 114.363315] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:350566330worker2 # [ 114.376073] nix-grpc-daemon[1227]: time=2026-09-23T05:13:44.541Z level=INFO msg="All 2 paths already cached"6331client # copying 1 paths to 'lb:50051'...6332client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...6333worker2 # [ 114.385812] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:350566334worker2 # [ 114.394382] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)6335worker2 # [ 114.499358] nix-daemon[851]: accepted connection from pid 1336, user nix-grpc-daemon (trusted)6336worker2 # [ 114.514654] nix-daemon[851]: reaped child process 1339, status = succeeded6337lb # [ 114.533674] niks3-server[1118]: 2026/09/23 05:13:44 INFO Received uploads request method=POST path=/api/pending_closures6338worker2 # [ 114.530158] nix-grpc-daemon[1227]: time=2026-09-23T05:13:44.695Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6339worker2 # [ 114.531699] nix-grpc-daemon[1227]: time=2026-09-23T05:13:44.697Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"6340lb # [ 114.565155] niks3-server[1118]: 2026/09/23 05:13:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6341worker2 # [ 114.560571] nix-grpc-daemon[1227]: time=2026-09-23T05:13:44.725Z level=INFO msg="Uploading 1 narinfos"6342lb # [ 114.573400] niks3-server[1118]: 2026/09/23 05:13:44 INFO Signed narinfos id=47 count=16343lb # [ 114.578898] niks3-server[1118]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls6344lb # [ 114.588648] niks3-server[1118]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst6345lb # [ 114.597846] niks3-server[1118]: 2026/09/23 05:13:44 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6346lb # [ 114.605456] niks3-server[1118]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo6347lb # [ 114.610172] niks3-server[1118]: 2026/09/23 05:13:44 INFO Completed upload id=476348worker2 # [ 114.601140] nix-grpc-daemon[1227]: time=2026-09-23T05:13:44.766Z level=INFO msg="Upload complete. (193ms)"6349worker2 # [ 114.606495] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35056 duration_s=0 paths=1 nar_bytes_in=7686350worker2 # [ 114.612166] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:350566351worker2 # [ 114.624323] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)6352worker2 # [ 114.632991] nix-daemon[851]: reaped child process 1334, status = succeeded6353worker2 # [ 114.638284] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)6354client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...6355client # farm-top> worker2: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6356lb # [ 114.878492] envoy[903]: conn peer=[2001:db8:1::3]:55452 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1726 tx=592 ms=93746357client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6358client # farm-top> LOG-farm-top-on-standby6359worker2 # [ 115.457924] nix-daemon[851]: reaped child process 1343, status = succeeded6360worker2 # [ 115.540815] nix-daemon[851]: accepted connection from pid 1359, user nix-grpc-daemon (trusted)6361worker2 # [ 115.554087] nix-daemon[851]: reaped child process 1362, status = succeeded6362lb # [ 115.572786] niks3-server[1118]: 2026/09/23 05:13:45 INFO Received uploads request method=POST path=/api/pending_closures6363worker2 # [ 115.571672] nix-grpc-daemon[1227]: time=2026-09-23T05:13:45.736Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6364worker2 # [ 115.574107] nix-grpc-daemon[1227]: time=2026-09-23T05:13:45.738Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"6365lb # [ 115.616882] niks3-server[1118]: 2026/09/23 05:13:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6366lb # [ 115.625126] niks3-server[1118]: 2026/09/23 05:13:45 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst6367worker2 # [ 115.621255] nix-grpc-daemon[1227]: time=2026-09-23T05:13:45.786Z level=INFO msg="Uploading 1 narinfos"6368lb # [ 115.632767] niks3-server[1118]: 2026/09/23 05:13:45 INFO Signed narinfos id=48 count=16369lb # [ 115.635804] niks3-server[1118]: 2026/09/23 05:13:45 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6370lb # [ 115.644637] niks3-server[1118]: 2026/09/23 05:13:45 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls6371lb # [ 115.652753] niks3-server[1118]: 2026/09/23 05:13:45 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6372lb # [ 115.657806] niks3-server[1118]: 2026/09/23 05:13:45 INFO Completed upload id=486373lb # [ 115.660658] niks3-server[1118]: 2026/09/23 05:13:45 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo6374worker2 # [ 115.647946] nix-grpc-daemon[1227]: time=2026-09-23T05:13:45.813Z level=INFO msg="Upload complete. (202ms)"6375worker2 # [ 115.656552] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35056 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=16376worker2 # [ 115.671303] nix-daemon[851]: reaped child process 1341, status = succeeded6377client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&2, in 4.66 seconds)6378client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath6379client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath, in 0.11 seconds)6380worker1 # [ 115.868657] systemd[1]: Listening on Nix worker-protocol over gRPC.6381worker1 # [ 115.875902] systemd[1]: Starting Nix worker-protocol over gRPC...6382worker1 # [ 115.963808] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:45Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6383worker1 # [ 115.965871] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:45Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6384worker1 # [ 115.969454] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:45Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16385worker1 # [ 115.973975] nix-grpc-daemon[1528]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6386worker1 # [ 115.975345] nix-grpc-daemon[1528]: E0000 00:00:1790140425.967046 1528 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6387worker1 # [ 115.981064] nix-grpc-daemon[1528]: E0000 00:00:1790140425.972513 1528 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6388worker1 # [ 115.984770] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:45Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6389worker1 # [ 115.988697] systemd[1]: Started Nix worker-protocol over gRPC.6390worker1 # [ 115.991070] nix-grpc-daemon[1528]: E0000 00:00:1790140425.984003 1570 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6391worker1 # [ 116.004405] nix-grpc-daemon[1528]: E0000 00:00:1790140425.997987 1570 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6392worker1 # [ 116.008059] nix-grpc-daemon[1528]: E0000 00:00:1790140426.000407 1570 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6393lb: must succeed: curl -sf localhost:9901/clusters6394worker1 # [ 116.020069] nix-grpc-daemon[1528]: E0000 00:00:1790140426.013598 1570 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6395worker1 # [ 116.031576] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:46Z level=info event=scheduler_connected addr=lb:500516396worker2 # [ 116.019414] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:46Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:329086397worker2 # [ 116.023469] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:46Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06398lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6399worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6400worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6401lb: must succeed: curl -sf localhost:9901/clusters6402lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6403lb: must succeed: curl -sf localhost:9901/clusters6404lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6405worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6406worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6407(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 11.41 seconds)6408subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6409client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs6410client # [ 116.269856] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs.6411client # Running as unit: rs.service; invocation ID: 185da48d4d62436caf6ecb621277bdb96412client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs, in 0.06 seconds)6413worker2 # [ 116.395358] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:350566414client # [ 116.412877] nix[1355]: don't know how to build these paths:6415client # [ 116.413925] nix[1355]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6416worker2 # [ 116.408269] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:46Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06417worker2 # [ 116.413686] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:46Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.4:50051 clients=16418worker2 # [ 116.452658] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:350566419worker2 # [ 116.465069] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:350566420client # [ 116.487376] nix[1355]: copying 1 paths to 'lb:50051'...6421worker2 # [ 116.477443] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:350566422client # [ 116.489609] nix[1355]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6423worker2 # [ 116.489777] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)6424worker2 # [ 116.586068] nix-daemon[851]: accepted connection from pid 1375, user nix-grpc-daemon (trusted)6425worker2 # [ 116.598862] nix-daemon[851]: reaped child process 1378, status = succeeded6426lb # [ 116.617700] niks3-server[1118]: 2026/09/23 05:13:46 INFO Received uploads request method=POST path=/api/pending_closures6427worker2 # [ 116.614536] nix-grpc-daemon[1227]: time=2026-09-23T05:13:46.779Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6428worker2 # [ 116.616598] nix-grpc-daemon[1227]: time=2026-09-23T05:13:46.781Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6429lb # [ 116.651151] niks3-server[1118]: 2026/09/23 05:13:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6430lb # [ 116.658472] niks3-server[1118]: 2026/09/23 05:13:46 INFO Signed narinfos id=49 count=16431worker2 # [ 116.650966] nix-grpc-daemon[1227]: time=2026-09-23T05:13:46.816Z level=INFO msg="Uploading 1 narinfos"6432lb # [ 116.664124] niks3-server[1118]: 2026/09/23 05:13:46 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6433lb # [ 116.669248] niks3-server[1118]: 2026/09/23 05:13:46 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6434lb # [ 116.682071] niks3-server[1118]: 2026/09/23 05:13:46 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6435lb # [ 116.689880] niks3-server[1118]: 2026/09/23 05:13:46 INFO Completed upload id=496436worker2 # [ 116.678396] nix-grpc-daemon[1227]: time=2026-09-23T05:13:46.843Z level=INFO msg="Upload complete. (177ms)"6437lb # [ 116.692822] niks3-server[1118]: 2026/09/23 05:13:46 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6438worker2 # [ 116.684435] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35056 duration_s=0 paths=1 nar_bytes_in=4166439worker2 # [ 116.690334] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:350566440worker2 # [ 116.701154] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)6441worker2 # [ 116.711092] nix-daemon[851]: reaped child process 1373, status = succeeded6442worker2 # [ 116.714449] nix-daemon[851]: accepted connection from pid 1166, user nix-grpc-daemon (trusted)6443client # [ 116.725463] nix[1355]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6444lb: must succeed: curl -sf localhost:9901/clusters6445lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6446lb: must succeed: curl -sf localhost:9901/clusters6447lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6448lb: must succeed: curl -sf localhost:9901/clusters6449lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6450lb: must succeed: curl -sf localhost:9901/clusters6451lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6452worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6453worker1 # [ 118.945036] systemd-journald[284]: Received client request to rotate journal, rotating.6454worker1 # [ 118.965522] systemd-journald[284]: Vacuuming done, freed 0B of archived journals from /var/log/journal/a9dd989bdecd4fd0a3bc02e8cfb9d3ff.6455worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.07 seconds)6456worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service6457worker2 # [ 118.862906] systemd[1]: Stopping Nix worker-protocol over gRPC...6458worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.03 seconds)6459client: must succeed: sleep 5.06460worker2 # [ 119.202939] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:49Z level=info event=draining6461client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6462worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6463worker2 # [ 124.134827] nix-daemon[851]: reaped child process 1382, status = succeeded6464worker2 # [ 124.210647] nix-daemon[851]: accepted connection from pid 1423, user nix-grpc-daemon (trusted)6465worker2 # [ 124.221533] nix-daemon[851]: reaped child process 1426, status = succeeded6466lb # [ 124.241728] niks3-server[1118]: 2026/09/23 05:13:54 INFO Received uploads request method=POST path=/api/pending_closures6467worker2 # [ 124.238643] nix-grpc-daemon[1227]: time=2026-09-23T05:13:54.403Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6468worker2 # [ 124.241336] nix-grpc-daemon[1227]: time=2026-09-23T05:13:54.405Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6469lb # [ 124.279657] niks3-server[1118]: 2026/09/23 05:13:54 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6470lb # [ 124.294399] niks3-server[1118]: 2026/09/23 05:13:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6471worker2 # [ 124.288939] nix-grpc-daemon[1227]: time=2026-09-23T05:13:54.454Z level=INFO msg="Uploading 1 narinfos"6472lb # [ 124.301562] niks3-server[1118]: 2026/09/23 05:13:54 INFO Signed narinfos id=50 count=16473lb # [ 124.304363] niks3-server[1118]: 2026/09/23 05:13:54 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6474lb # [ 124.315115] niks3-server[1118]: 2026/09/23 05:13:54 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6475lb # [ 124.320093] niks3-server[1118]: 2026/09/23 05:13:54 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6476lb # [ 124.324352] niks3-server[1118]: 2026/09/23 05:13:54 INFO Completed upload id=506477worker2 # [ 124.312971] nix-grpc-daemon[1227]: time=2026-09-23T05:13:54.478Z level=INFO msg="Upload complete. (185ms)"6478lb # [ 124.328148] niks3-server[1118]: 2026/09/23 05:13:54 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6479worker2 # [ 124.321621] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35056 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16480worker2 # [ 124.334982] nix-daemon[851]: reaped child process 1380, status = succeeded6481client # [ 124.362295] systemd[1]: rs.service: Deactivated successfully.6482client # [ 124.364785] systemd[1]: rs.service: Consumed 123ms CPU time over 8.091s wall clock time, 43.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.6483lb # [ 124.879339] envoy[903]: conn peer=[2001:db8:1::1]:33630 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9586 tx=4396 ms=45036484lb # [ 124.883770] envoy[903]: conn peer=[2001:db8:1::1]:55746 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2956 tx=1521 ms=79356485worker2 # [ 125.205148] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:55Z level=info event=signal_exit6486lb # [ 125.611128] niks3-server[1118]: 2026/09/23 05:13:55 INFO lead: released remote=192.168.1.4:512986487lb # [ 125.990453] niks3-server[1118]: 2026/09/23 05:13:56 INFO lead: acquired remote=192.168.1.3:497846488worker1 # [ 125.997254] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:55Z level=info event=scheduler_take_over6489worker1 # [ 126.614433] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:56Z level=info event=scheduler_restarting addr=lb:500516490worker2 # [ 126.604235] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:56Z level=info event=scheduler_restarting6491worker2 # [ 126.607540] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:56Z level=info event=scheduler_restarting addr=lb:500516492worker2 # [ 126.611230] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:56Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:329086493worker2 # [ 126.615816] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:56Z level=info event=worker_gone addr=192.168.1.3:500516494worker2 # [ 126.619189] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:56Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:329086495worker2 # [ 126.622810] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:56Z level=info event=worker_gone addr=192.168.1.4:500516496worker2 # [ 126.643640] nix-daemon[851]: reaped child process 1219, status = succeeded6497worker1 # [ 126.815575] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:56Z level=info event=scheduler_connected addr=lb:500516498worker1 # [ 126.820123] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6499worker2 # [ 126.813605] nix-grpc-daemon[1166]: ts=2026-09-23T05:13:56Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6500worker1 # [ 127.021618] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:57Z level=info event=scheduler_connected addr=lb:500516501worker1 # [ 127.028877] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:57Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:421466502worker1 # [ 127.033747] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:57Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06503worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.18 seconds)6504client: waiting for success: ! systemctl is-active rs6505worker2 # [ 127.278642] nix-grpc-daemon[1166]: E0000 00:00:1790140437.443721 1212 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true6506worker2 # [ 127.283591] nix-grpc-daemon[1166]: E0000 00:00:1790140437.443909 1212 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6507client: (finished: waiting for success: ! systemctl is-active rs, in 0.04 seconds)6508client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6509client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.03 seconds)6510client: must fail: journalctl -u rs | grep -q 'reconnecting'6511client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.01 seconds)6512lb: must succeed: curl -sf localhost:9901/clusters6513lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6514worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6515worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6516worker2 # [ 128.136487] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6517worker2 # [ 128.140395] systemd[1]: Stopped Nix worker-protocol over gRPC.6518worker2 # [ 128.143833] systemd[1]: nix-grpc-daemon.service: Consumed 1.265s CPU time over 35.042s wall clock time, 38.5M memory peak, 63.7K incoming IP traffic, 160.2K outgoing IP traffic.6519worker2 # [ 128.158184] systemd[1]: Starting scheduler-token.service...6520worker2 # [ 128.204057] systemd[1]: scheduler-token.service: Deactivated successfully.6521worker2 # [ 128.206172] systemd[1]: Finished scheduler-token.service.6522worker2 # [ 128.206970] systemd[1]: scheduler-token.service: Consumed 24ms CPU time over 47ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6523worker2 # [ 128.210801] systemd[1]: Starting Nix worker-protocol over gRPC...6524worker2 # [ 128.285749] nix-grpc-daemon[1436]: ts=2026-09-23T05:13:58Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6525worker2 # [ 128.287774] nix-grpc-daemon[1436]: ts=2026-09-23T05:13:58Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6526worker2 # [ 128.290921] nix-grpc-daemon[1436]: ts=2026-09-23T05:13:58Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16527worker2 # [ 128.296054] nix-grpc-daemon[1436]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6528worker2 # [ 128.297420] nix-grpc-daemon[1436]: E0000 00:00:1790140438.460402 1436 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6529worker2 # [ 128.302731] nix-grpc-daemon[1436]: E0000 00:00:1790140438.465940 1436 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6530worker2 # [ 128.306281] nix-grpc-daemon[1436]: ts=2026-09-23T05:13:58Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6531worker2 # [ 128.307922] systemd[1]: Started Nix worker-protocol over gRPC.6532worker2 # [ 128.311068] nix-grpc-daemon[1436]: E0000 00:00:1790140438.475957 1479 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6533worker2 # [ 128.322709] nix-grpc-daemon[1436]: E0000 00:00:1790140438.488021 1479 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6534worker2 # [ 128.327056] nix-grpc-daemon[1436]: E0000 00:00:1790140438.490383 1479 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6535worker2 # [ 128.336774] nix-grpc-daemon[1436]: E0000 00:00:1790140438.502089 1479 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6536worker2 # [ 128.346237] nix-grpc-daemon[1436]: ts=2026-09-23T05:13:58Z level=info event=scheduler_connected addr=lb:500516537worker1 # [ 128.370109] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:58Z level=info event=oidc_jwks provider=mock keys=16538worker1 # [ 128.373472] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:58Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:421466539worker1 # [ 128.377451] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:58Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06540lb: must succeed: curl -sf localhost:9901/clusters6541lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6542worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6543worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6544(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.32 seconds)6545subtest: scheduler restart mid-build: build finishes, no double build6546client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid6547client # [ 128.590660] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid.6548client # Running as unit: mid.service; invocation ID: 943a0ef7bc6f4471b5baabc2f3f839e86549client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid, in 0.06 seconds)6550worker1 # [ 128.729624] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:421526551worker1 # [ 128.741746] nix-daemon[864]: accepted connection from pid 1528, user nix-grpc-daemon (trusted)6552client # [ 128.751922] nix[1391]: don't know how to build these paths:6553client # [ 128.752849] nix[1391]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6554worker1 # [ 128.761649] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:58Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06555worker1 # [ 128.766808] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:58Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16556worker1 # [ 128.811358] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421526557worker1 # [ 128.823189] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 13 ms (attempt 1/5)6558worker1 # [ 128.843560] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 49 ms (attempt 2/5)6559worker1 # [ 128.894612] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 82 ms (attempt 3/5)6560worker1 # [ 128.978641] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 91 ms (attempt 4/5)6561worker1 # [ 129.072374] nix-grpc-daemon[1528]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6562worker1 # [ 129.076360] nix-grpc-daemon[1528]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6563worker1 # [ 129.092650] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421526564client # [ 129.097914] nix[1391]: copying 1 paths to 'lb:50051'...6565client # [ 129.102185] nix[1391]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6566worker1 # [ 129.105501] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421526567worker1 # [ 129.116382] nix-daemon[864]: accepted connection from pid 1528, user nix-grpc-daemon (trusted)6568worker1 # [ 129.136656] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:59Z level=info event=niks3_push_started pid=16576569worker1 # [ 129.237106] nix-grpc-daemon[1657]: time=2026-09-23T05:13:59.230Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6570worker1 # [ 129.324585] nix-daemon[864]: accepted connection from pid 1670, user nix-grpc-daemon (trusted)6571worker1 # [ 129.337101] nix-daemon[864]: reaped child process 1673, status = succeeded6572lb # [ 129.342748] niks3-server[1118]: 2026/09/23 05:13:59 INFO Received uploads request method=POST path=/api/pending_closures6573worker1 # [ 129.353324] nix-grpc-daemon[1657]: time=2026-09-23T05:13:59.346Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6574worker1 # [ 129.354902] nix-grpc-daemon[1657]: time=2026-09-23T05:13:59.348Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6575lb # [ 129.381996] niks3-server[1118]: 2026/09/23 05:13:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6576lb # [ 129.388952] niks3-server[1118]: 2026/09/23 05:13:59 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6577lb # [ 129.402183] niks3-server[1118]: 2026/09/23 05:13:59 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6578lb # [ 129.410422] niks3-server[1118]: 2026/09/23 05:13:59 INFO Signed narinfos id=51 count=16579worker1 # [ 129.415411] nix-grpc-daemon[1657]: time=2026-09-23T05:13:59.408Z level=INFO msg="Uploading 1 narinfos"6580lb # [ 129.424743] niks3-server[1118]: 2026/09/23 05:13:59 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6581lb # [ 129.427900] niks3-server[1118]: 2026/09/23 05:13:59 INFO Completed upload id=516582worker1 # [ 129.430455] nix-grpc-daemon[1657]: time=2026-09-23T05:13:59.423Z level=INFO msg="Upload complete. (190ms)"6583worker1 # [ 129.432931] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42152 duration_s=0 paths=1 nar_bytes_in=4246584lb # [ 129.431678] niks3-server[1118]: 2026/09/23 05:13:59 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo6585worker1 # [ 129.436563] nix-grpc-daemon[1528]: ts=2026-09-23T05:13:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421526586worker1 # [ 129.444243] nix-daemon[864]: accepted connection from pid 1528, user nix-grpc-daemon (trusted)6587worker1 # [ 129.446610] nix-daemon[864]: reaped child process 1655, status = succeeded6588worker1 # [ 129.452739] nix-daemon[864]: accepted connection from pid 1528, user nix-grpc-daemon (trusted)6589client # [ 129.448668] nix[1391]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6590lb: must succeed: curl -sf localhost:9901/clusters6591lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6592lb: must succeed: curl -sf localhost:9901/clusters6593lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6594worker1: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6595worker1 # [ 131.069361] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1528 (nix-grpc-daemon) on client request.6596worker1 # [ 131.071915] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6597lb # [ 131.074304] niks3-server[1118]: 2026/09/23 05:14:01 INFO lead: released remote=192.168.1.3:497846598client # [ 131.077626] nix[1391]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6599worker2 # [ 131.066675] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:01Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6600worker1 # [ 131.087301] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6601client # [ 131.081571] nix[1391]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6602worker1 # [ 131.088761] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6603worker1 # [ 131.092128] systemd[1]: nix-grpc-daemon.service: Consumed 371ms CPU time over 15.212s wall clock time, 35.5M memory peak, 15.4K incoming IP traffic, 27.5K outgoing IP traffic.6604worker1 # [ 131.106084] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6605worker1 # [ 131.107651] systemd[1]: Starting Nix worker-protocol over gRPC...6606worker1 # [ 131.124368] nix-daemon[864]: reaped child process 1675, status = succeeded6607worker1 # [ 131.131703] nix-daemon[864]: reaped child process 1647, status = succeeded6608worker1 # [ 131.141063] nix-daemon[1677]: unexpected Nix daemon error: error: interrupted by the user6609worker1 # [ 131.150736] nix-daemon[864]: reaped child process 1677, status = failed with exit code 16610worker1 # [ 131.196809] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6611worker1 # [ 131.198778] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6612worker1 # [ 131.202026] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16613worker1 # [ 131.206468] nix-grpc-daemon[1714]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6614worker1 # [ 131.207880] nix-grpc-daemon[1714]: E0000 00:00:1790140441.199839 1714 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6615worker1 # [ 131.215055] nix-grpc-daemon[1714]: E0000 00:00:1790140441.205290 1714 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6616worker1 # [ 131.218521] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6617worker1 # [ 131.220170] systemd[1]: Started Nix worker-protocol over gRPC.6618lb # [ 131.222076] niks3-server[1118]: 2026/09/23 05:14:01 INFO lead: acquired remote=192.168.1.3:491526619worker1 # [ 131.225078] nix-grpc-daemon[1714]: E0000 00:00:1790140441.217304 1756 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6620worker1 # [ 131.229070] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=info event=scheduler_take_over6621worker1: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.20 seconds)6622client: must succeed: sleep 5.06623worker1 # [ 131.237963] nix-grpc-daemon[1714]: E0000 00:00:1790140441.231542 1756 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6624worker1 # [ 131.241150] nix-grpc-daemon[1714]: E0000 00:00:1790140441.233857 1756 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6625worker1 # [ 131.252192] nix-grpc-daemon[1714]: E0000 00:00:1790140441.245733 1756 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6626worker1 # [ 131.261432] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=info event=scheduler_connected addr=lb:500516627worker1 # [ 131.268107] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:421666628worker1 # [ 131.269659] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06629worker2 # [ 131.267545] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:01Z level=info event=scheduler_connected addr=lb:500516630worker1 # [ 131.285822] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=info event=oidc_jwks provider=mock keys=16631worker1 # [ 131.287228] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:421666632worker1 # [ 131.288911] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06633worker1 # [ 131.589309] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06634worker1 # [ 131.591682] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16635worker1 # [ 131.639871] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421726636worker1 # [ 131.645073] nix-daemon[864]: accepted connection from pid 1714, user nix-grpc-daemon (trusted)6637worker1 # [ 131.658698] nix-daemon[864]: accepted connection from pid 1714, user nix-grpc-daemon (trusted)6638client # [ 131.669097] nix[1391]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6639worker1 # [ 131.673431] nix-daemon[864]: accepted connection from pid 1714, user nix-grpc-daemon (trusted)6640lb # [ 134.879655] envoy[903]: conn peer=[2001:db8:1::4]:49412 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=36750 tx=2023 ms=335676641lb # [ 134.883635] envoy[903]: conn peer=[2001:db8:1::3]:34040 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1036 tx=393 ms=150556642client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6643client: waiting for success: ! systemctl is-active mid6644worker1 # [ 136.493942] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:06Z level=info event=niks3_push_started pid=17986645worker1 # [ 136.511761] nix-daemon[864]: reaped child process 1773, status = succeeded6646worker1 # [ 136.589877] nix-grpc-daemon[1798]: time=2026-09-23T05:14:06.583Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6647worker1 # [ 136.675361] nix-daemon[864]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6648worker1 # [ 136.688773] nix-daemon[864]: reaped child process 1814, status = succeeded6649lb # [ 136.694845] niks3-server[1118]: 2026/09/23 05:14:06 INFO Received uploads request method=POST path=/api/pending_closures6650worker1 # [ 136.709261] nix-grpc-daemon[1798]: time=2026-09-23T05:14:06.702Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6651worker1 # [ 136.710787] nix-grpc-daemon[1798]: time=2026-09-23T05:14:06.704Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6652lb # [ 136.749780] niks3-server[1118]: 2026/09/23 05:14:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6653worker1 # [ 136.760303] nix-grpc-daemon[1798]: time=2026-09-23T05:14:06.753Z level=INFO msg="Uploading 1 narinfos"6654lb # [ 136.759132] niks3-server[1118]: 2026/09/23 05:14:06 INFO Signed narinfos id=52 count=16655lb # [ 136.766086] niks3-server[1118]: 2026/09/23 05:14:06 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6656lb # [ 136.777259] niks3-server[1118]: 2026/09/23 05:14:06 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6657lb # [ 136.781978] niks3-server[1118]: 2026/09/23 05:14:06 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6658lb # [ 136.787067] niks3-server[1118]: 2026/09/23 05:14:06 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6659lb # [ 136.791073] niks3-server[1118]: 2026/09/23 05:14:06 INFO Completed upload id=526660worker1 # [ 136.793091] nix-grpc-daemon[1798]: time=2026-09-23T05:14:06.786Z level=INFO msg="Upload complete. (200ms)"6661lb # [ 136.794470] niks3-server[1118]: 2026/09/23 05:14:06 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6662worker1 # [ 136.795807] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42172 duration_s=5 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16663worker1 # [ 136.805353] nix-daemon[864]: reaped child process 1771, status = succeeded6664client # [ 136.827146] systemd[1]: mid.service: Deactivated successfully.6665client # [ 136.829785] systemd[1]: mid.service: Consumed 133ms CPU time over 8.236s wall clock time, 41.4M memory peak, 8.7K incoming IP traffic, 10.4K outgoing IP traffic.6666client: (finished: waiting for success: ! systemctl is-active mid, in 1.08 seconds)6667client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6668client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.03 seconds)6669(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 9.00 seconds)6670subtest: interrupting the client stops the build on the worker6671client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr6672client # [ 137.570260] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr.6673client # Running as unit: intr.service; invocation ID: 99aa8c6331174cc1bb55c2e836af926f6674client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr, in 0.03 seconds)6675worker2 # [ 137.686099] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:374506676worker2 # [ 137.697092] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6677client # [ 137.720224] nix[1427]: don't know how to build these paths:6678client # [ 137.721137] nix[1427]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6679worker1 # [ 137.730369] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:07Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06680worker1 # [ 137.732445] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:07Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=16681worker2 # [ 137.760451] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374506682worker2 # [ 137.773178] nix-grpc-daemon[1436]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 19 ms (attempt 1/5)6683worker2 # [ 137.799483] nix-grpc-daemon[1436]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 19 ms (attempt 2/5)6684worker2 # [ 137.820475] nix-grpc-daemon[1436]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 203 ms (attempt 3/5)6685worker2 # [ 138.031558] nix-grpc-daemon[1436]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 515 ms (attempt 4/5)6686worker2 # [ 138.549246] nix-grpc-daemon[1436]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6687worker2 # [ 138.552741] nix-grpc-daemon[1436]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6688worker2 # [ 138.570667] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374506689client # [ 138.592648] nix[1427]: copying 1 paths to 'lb:50051'...6690client # [ 138.596142] nix[1427]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6691worker2 # [ 138.585833] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374506692worker2 # [ 138.594664] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6693worker2 # [ 138.619219] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:08Z level=info event=niks3_push_started pid=15256694worker2 # [ 138.740081] nix-grpc-daemon[1525]: time=2026-09-23T05:14:08.904Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""6695worker2 # [ 138.829639] nix-daemon[851]: accepted connection from pid 1541, user nix-grpc-daemon (trusted)6696worker2 # [ 138.842063] nix-daemon[851]: reaped child process 1544, status = succeeded6697lb # [ 138.862175] niks3-server[1118]: 2026/09/23 05:14:08 INFO Received uploads request method=POST path=/api/pending_closures6698worker2 # [ 138.859059] nix-grpc-daemon[1525]: time=2026-09-23T05:14:09.024Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6699worker2 # [ 138.860888] nix-grpc-daemon[1525]: time=2026-09-23T05:14:09.025Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6700lb # [ 138.903603] niks3-server[1118]: 2026/09/23 05:14:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6701lb # [ 138.910556] niks3-server[1118]: 2026/09/23 05:14:08 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6702lb # [ 138.916491] niks3-server[1118]: 2026/09/23 05:14:08 INFO Signed narinfos id=53 count=16703worker2 # [ 138.908572] nix-grpc-daemon[1525]: time=2026-09-23T05:14:09.073Z level=INFO msg="Uploading 1 narinfos"6704lb # [ 138.929535] niks3-server[1118]: 2026/09/23 05:14:08 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6705lb # [ 138.937094] niks3-server[1118]: 2026/09/23 05:14:08 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6706lb # [ 138.947878] niks3-server[1118]: 2026/09/23 05:14:08 INFO Completed upload id=536707worker2 # [ 138.936558] nix-grpc-daemon[1525]: time=2026-09-23T05:14:09.101Z level=INFO msg="Upload complete. (194ms)"6708lb # [ 138.950349] niks3-server[1118]: 2026/09/23 05:14:08 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6709worker2 # [ 138.941760] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=0 paths=1 nar_bytes_in=4246710worker2 # [ 138.949090] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374506711worker2 # [ 138.956831] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6712worker2 # [ 138.966388] nix-daemon[851]: reaped child process 1523, status = succeeded6713worker2 # [ 138.970604] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6714client # [ 138.982324] nix[1427]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6715client: must succeed: systemctl kill -s INT intr6716worker2 # [ 140.973263] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:11Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6717client # [ 140.981984] nix[1427]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6718client # [ 140.990883] systemd[1]: intr.service: Sent signal SIGINT to main process 1427 (nix) on client request.6719client # [ 140.994365] nix[1427]: error: interrupted by the user6720client: (finished: must succeed: systemctl kill -s INT intr, in 0.05 seconds)6721worker2 # [ 140.990114] nix-daemon[851]: reaped child process 1546, status = succeeded6722worker2 # [ 140.991320] nix-daemon[1548]: unexpected Nix daemon error: error: interrupted by the user6723worker2 # [ 140.995287] nix-daemon[851]: reaped child process 1548, status = failed with exit code 16724client # [ 141.022195] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6725client # [ 141.025101] systemd[1]: intr.service: Failed with result 'exit-code'.6726client # [ 141.028674] systemd[1]: intr.service: Consumed 116ms CPU time over 3.454s wall clock time, 41.4M memory peak, 6.4K incoming IP traffic, 8.2K outgoing IP traffic.6727(finished: subtest: interrupting the client stops the build on the worker, in 3.52 seconds)6728subtest: a deploy mid-build drains: the build finishes, the new generation takes over6729client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw6730client # [ 141.104883] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw.6731client # Running as unit: sw.service; invocation ID: fd00aaf00ca744e99b21ada953ce1ef16732client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw, in 0.04 seconds)6733client # [ 141.224344] nix[1448]: don't know how to build these paths:6734client # [ 141.225225] nix[1448]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6735worker1 # [ 141.226939] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:421726736worker1 # [ 141.237245] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:11Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06737worker1 # [ 141.242723] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:11Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=16738worker1 # [ 141.281820] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421726739worker1 # [ 141.292774] nix-grpc-daemon[1714]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 55 ms (attempt 1/5)6740worker1 # [ 141.355697] nix-grpc-daemon[1714]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 30 ms (attempt 2/5)6741worker1 # [ 141.393983] nix-grpc-daemon[1714]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 359 ms (attempt 3/5)6742worker1 # [ 141.755281] nix-grpc-daemon[1714]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 751 ms (attempt 4/5)6743worker1 # [ 142.515922] nix-grpc-daemon[1714]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6744worker1 # [ 142.519340] nix-grpc-daemon[1714]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6745worker1 # [ 142.536585] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421726746client # [ 142.544640] nix[1448]: copying 1 paths to 'lb:50051'...6747client # [ 142.548617] nix[1448]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6748worker1 # [ 142.552105] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421726749worker1 # [ 142.563721] nix-daemon[864]: accepted connection from pid 1714, user nix-grpc-daemon (trusted)6750worker1 # [ 142.663889] nix-daemon[864]: accepted connection from pid 1854, user nix-grpc-daemon (trusted)6751worker1 # [ 142.676791] nix-daemon[864]: reaped child process 1857, status = succeeded6752lb # [ 142.682987] niks3-server[1118]: 2026/09/23 05:14:12 INFO Received uploads request method=POST path=/api/pending_closures6753worker1 # [ 142.693574] nix-grpc-daemon[1798]: time=2026-09-23T05:14:12.686Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6754worker1 # [ 142.696072] nix-grpc-daemon[1798]: time=2026-09-23T05:14:12.688Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6755lb # [ 142.713592] niks3-server[1118]: 2026/09/23 05:14:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6756worker1 # [ 142.727671] nix-grpc-daemon[1798]: time=2026-09-23T05:14:12.720Z level=INFO msg="Uploading 1 narinfos"6757lb # [ 142.725163] niks3-server[1118]: 2026/09/23 05:14:12 INFO Signed narinfos id=54 count=16758lb # [ 142.729462] niks3-server[1118]: 2026/09/23 05:14:12 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6759lb # [ 142.735285] niks3-server[1118]: 2026/09/23 05:14:12 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6760lb # [ 142.747622] niks3-server[1118]: 2026/09/23 05:14:12 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6761lb # [ 142.756418] niks3-server[1118]: 2026/09/23 05:14:12 INFO Completed upload id=546762worker1 # [ 142.759167] nix-grpc-daemon[1798]: time=2026-09-23T05:14:12.752Z level=INFO msg="Upload complete. (183ms)"6763lb # [ 142.758616] niks3-server[1118]: 2026/09/23 05:14:12 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6764worker1 # [ 142.763951] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42172 duration_s=0 paths=1 nar_bytes_in=4166765worker1 # [ 142.770316] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421726766worker1 # [ 142.781135] nix-daemon[864]: accepted connection from pid 1714, user nix-grpc-daemon (trusted)6767worker1 # [ 142.790937] nix-daemon[864]: reaped child process 1852, status = succeeded6768worker1 # [ 142.795410] nix-daemon[864]: accepted connection from pid 1714, user nix-grpc-daemon (trusted)6769client # [ 142.792068] nix[1448]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6770worker1: must succeed: systemctl reload nix-grpc-daemon.service6771worker1 # [ 144.618241] systemd[1]: Reloading Nix worker-protocol over gRPC...6772worker1 # [ 144.643826] systemd[1]: Reloaded Nix worker-protocol over gRPC.6773worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.06 seconds)6774worker1: must succeed: pgrep -f 'read -t [9]0 x'6775worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.04 seconds)6776client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&26777worker2 # [ 144.813185] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:374506778client # don't know how to build these paths:6779client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6780worker1 # [ 144.836646] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421726781worker1 # [ 144.855683] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:14Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06782worker1 # [ 144.857636] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:14Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16783lb # [ 144.879814] envoy[903]: conn peer=[2001:db8:1::1]:60252 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3921 tx=1939 ms=80836784lb # [ 144.883551] envoy[903]: conn peer=[2001:db8:1::1]:55288 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2967 tx=830 ms=33186785worker2 # [ 144.891503] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374506786worker2 # [ 144.904835] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374506787client # copying 1 paths to 'lb:50051'...6788worker2 # [ 144.917504] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374506789client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6790worker2 # [ 144.938659] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6791worker2 # [ 145.040562] nix-daemon[851]: accepted connection from pid 1599, user nix-grpc-daemon (trusted)6792worker2 # [ 145.052655] nix-daemon[851]: reaped child process 1602, status = succeeded6793lb # [ 145.072648] niks3-server[1118]: 2026/09/23 05:14:15 INFO Received uploads request method=POST path=/api/pending_closures6794worker2 # [ 145.070506] nix-grpc-daemon[1525]: time=2026-09-23T05:14:15.235Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6795worker2 # [ 145.072030] nix-grpc-daemon[1525]: time=2026-09-23T05:14:15.235Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6796lb # [ 145.107898] niks3-server[1118]: 2026/09/23 05:14:15 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6797lb # [ 145.118148] niks3-server[1118]: 2026/09/23 05:14:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6798lb # [ 145.123529] niks3-server[1118]: 2026/09/23 05:14:15 INFO Signed narinfos id=55 count=16799worker2 # [ 145.116228] nix-grpc-daemon[1525]: time=2026-09-23T05:14:15.281Z level=INFO msg="Uploading 1 narinfos"6800lb # [ 145.134107] niks3-server[1118]: 2026/09/23 05:14:15 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6801lb # [ 145.146683] niks3-server[1118]: 2026/09/23 05:14:15 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6802lb # [ 145.152264] niks3-server[1118]: 2026/09/23 05:14:15 INFO Completed upload id=556803worker2 # [ 145.142297] nix-grpc-daemon[1525]: time=2026-09-23T05:14:15.307Z level=INFO msg="Upload complete. (192ms)"6804lb # [ 145.155423] niks3-server[1118]: 2026/09/23 05:14:15 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6805worker2 # [ 145.146304] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=0 paths=1 nar_bytes_in=5286806worker2 # [ 145.152156] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374506807worker2 # [ 145.160533] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6808worker2 # [ 145.169640] nix-daemon[851]: reaped child process 1597, status = succeeded6809client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6810worker2 # [ 145.176576] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6811client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6812worker1 # [ 145.218168] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:15Z level=info event=draining6813client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6814client # farm-a> LOG-farm-a-during-drain6815worker2 # [ 146.414136] nix-daemon[851]: reaped child process 1606, status = succeeded6816worker2 # [ 146.492925] nix-daemon[851]: accepted connection from pid 1622, user nix-grpc-daemon (trusted)6817worker2 # [ 146.505371] nix-daemon[851]: reaped child process 1625, status = succeeded6818lb # [ 146.525938] niks3-server[1118]: 2026/09/23 05:14:16 INFO Received uploads request method=POST path=/api/pending_closures6819worker2 # [ 146.522848] nix-grpc-daemon[1525]: time=2026-09-23T05:14:16.687Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6820worker2 # [ 146.524509] nix-grpc-daemon[1525]: time=2026-09-23T05:14:16.689Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6821lb # [ 146.569633] niks3-server[1118]: 2026/09/23 05:14:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6822worker2 # [ 146.563421] nix-grpc-daemon[1525]: time=2026-09-23T05:14:16.728Z level=INFO msg="Uploading 1 narinfos"6823lb # [ 146.574334] niks3-server[1118]: 2026/09/23 05:14:16 INFO Signed narinfos id=56 count=16824lb # [ 146.590372] niks3-server[1118]: 2026/09/23 05:14:16 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6825lb # [ 146.595810] niks3-server[1118]: 2026/09/23 05:14:16 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6826lb # [ 146.604585] niks3-server[1118]: 2026/09/23 05:14:16 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6827lb # [ 146.609335] niks3-server[1118]: 2026/09/23 05:14:16 INFO Completed upload id=566828worker2 # [ 146.597883] nix-grpc-daemon[1525]: time=2026-09-23T05:14:16.763Z level=INFO msg="Upload complete. (214ms)"6829lb # [ 146.610908] niks3-server[1118]: 2026/09/23 05:14:16 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6830lb # [ 146.613552] niks3-server[1118]: 2026/09/23 05:14:16 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6831worker2 # [ 146.605202] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16832worker1 # [ 146.623967] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:16Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06833worker1 # [ 146.627238] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:16Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16834worker2 # [ 146.619732] nix-daemon[851]: reaped child process 1604, status = succeeded6835worker2 # [ 146.654222] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374506836worker2 # [ 146.667433] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374506837worker2 # [ 146.681385] nix-grpc-daemon[1525]: time=2026-09-23T05:14:16.846Z level=INFO msg="All 1 paths already cached"6838client # copying 1 paths to 'lb:50051'...6839worker2 # [ 146.689973] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374506840client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6841worker2 # [ 146.709180] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6842worker2 # [ 146.808414] nix-daemon[851]: accepted connection from pid 1629, user nix-grpc-daemon (trusted)6843worker2 # [ 146.820781] nix-daemon[851]: reaped child process 1632, status = succeeded6844lb # [ 146.840748] niks3-server[1118]: 2026/09/23 05:14:16 INFO Received uploads request method=POST path=/api/pending_closures6845worker2 # [ 146.839357] nix-grpc-daemon[1525]: time=2026-09-23T05:14:17.004Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6846worker2 # [ 146.840902] nix-grpc-daemon[1525]: time=2026-09-23T05:14:17.006Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6847lb # [ 146.872109] niks3-server[1118]: 2026/09/23 05:14:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6848lb # [ 146.880322] niks3-server[1118]: 2026/09/23 05:14:16 INFO Signed narinfos id=57 count=16849worker2 # [ 146.872551] nix-grpc-daemon[1525]: time=2026-09-23T05:14:17.037Z level=INFO msg="Uploading 1 narinfos"6850lb # [ 146.889876] niks3-server[1118]: 2026/09/23 05:14:16 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6851lb # [ 146.896736] niks3-server[1118]: 2026/09/23 05:14:16 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6852lb # [ 146.904792] niks3-server[1118]: 2026/09/23 05:14:16 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6853lb # [ 146.912580] niks3-server[1118]: 2026/09/23 05:14:16 INFO Completed upload id=576854worker2 # [ 146.904114] nix-grpc-daemon[1525]: time=2026-09-23T05:14:17.069Z level=INFO msg="Upload complete. (182ms)"6855lb # [ 146.918539] niks3-server[1118]: 2026/09/23 05:14:16 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6856worker2 # [ 146.908077] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=0 paths=1 nar_bytes_in=6406857worker2 # [ 146.915266] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374506858worker2 # [ 146.923137] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6859worker2 # [ 146.933064] nix-daemon[851]: reaped child process 1627, status = succeeded6860client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6861worker2 # [ 146.937703] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6862client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6863client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6864client # farm-b> LOG-farm-b-during-drain6865worker2 # [ 147.777896] nix-daemon[851]: reaped child process 1636, status = succeeded6866worker2 # [ 147.859294] nix-daemon[851]: accepted connection from pid 1652, user nix-grpc-daemon (trusted)6867worker2 # [ 147.871162] nix-daemon[851]: reaped child process 1655, status = succeeded6868lb # [ 147.891890] niks3-server[1118]: 2026/09/23 05:14:17 INFO Received uploads request method=POST path=/api/pending_closures6869worker2 # [ 147.888210] nix-grpc-daemon[1525]: time=2026-09-23T05:14:18.053Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6870worker2 # [ 147.889821] nix-grpc-daemon[1525]: time=2026-09-23T05:14:18.055Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6871lb # [ 147.930419] niks3-server[1118]: 2026/09/23 05:14:17 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6872lb # [ 147.941110] niks3-server[1118]: 2026/09/23 05:14:17 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6873lb # [ 147.948096] niks3-server[1118]: 2026/09/23 05:14:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6874lb # [ 147.954167] niks3-server[1118]: 2026/09/23 05:14:17 INFO Signed narinfos id=58 count=16875worker2 # [ 147.945334] nix-grpc-daemon[1525]: time=2026-09-23T05:14:18.110Z level=INFO msg="Uploading 1 narinfos"6876lb # [ 147.957750] niks3-server[1118]: 2026/09/23 05:14:17 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6877lb # [ 147.969979] niks3-server[1118]: 2026/09/23 05:14:17 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6878lb # [ 147.974847] niks3-server[1118]: 2026/09/23 05:14:18 INFO Completed upload id=586879lb # [ 147.976558] niks3-server[1118]: 2026/09/23 05:14:18 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6880worker2 # [ 147.963491] nix-grpc-daemon[1525]: time=2026-09-23T05:14:18.128Z level=INFO msg="Upload complete. (219ms)"6881worker1 # [ 147.988357] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:17Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06882worker2 # [ 147.969736] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=16883worker1 # [ 147.990274] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:17Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16884worker2 # [ 147.985534] nix-daemon[851]: reaped child process 1634, status = succeeded6885worker2 # [ 148.018741] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374506886worker2 # [ 148.030996] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374506887worker2 # [ 148.043815] nix-grpc-daemon[1525]: time=2026-09-23T05:14:18.209Z level=INFO msg="All 2 paths already cached"6888client # copying 1 paths to 'lb:50051'...6889client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6890worker2 # [ 148.053158] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374506891worker2 # [ 148.061401] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6892worker2 # [ 148.160898] nix-daemon[851]: accepted connection from pid 1659, user nix-grpc-daemon (trusted)6893worker2 # [ 148.174148] nix-daemon[851]: reaped child process 1662, status = succeeded6894lb # [ 148.194144] niks3-server[1118]: 2026/09/23 05:14:18 INFO Received uploads request method=POST path=/api/pending_closures6895worker2 # [ 148.190209] nix-grpc-daemon[1525]: time=2026-09-23T05:14:18.355Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6896worker2 # [ 148.191657] nix-grpc-daemon[1525]: time=2026-09-23T05:14:18.357Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6897lb # [ 148.226931] niks3-server[1118]: 2026/09/23 05:14:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6898lb # [ 148.230672] niks3-server[1118]: 2026/09/23 05:14:18 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6899lb # [ 148.238492] niks3-server[1118]: 2026/09/23 05:14:18 INFO Signed narinfos id=59 count=16900worker2 # [ 148.229766] nix-grpc-daemon[1525]: time=2026-09-23T05:14:18.395Z level=INFO msg="Uploading 1 narinfos"6901lb # [ 148.242827] niks3-server[1118]: 2026/09/23 05:14:18 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6902lb # [ 148.256963] niks3-server[1118]: 2026/09/23 05:14:18 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6903worker2 # [ 148.251067] nix-grpc-daemon[1525]: time=2026-09-23T05:14:18.416Z level=INFO msg="Upload complete. (176ms)"6904lb # [ 148.262750] niks3-server[1118]: 2026/09/23 05:14:18 INFO Completed upload id=596905worker2 # [ 148.253738] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=0 paths=1 nar_bytes_in=7686906lb # [ 148.267912] niks3-server[1118]: 2026/09/23 05:14:18 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6907worker2 # [ 148.260813] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374506908worker2 # [ 148.271737] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6909worker2 # [ 148.274349] nix-daemon[851]: reaped child process 1657, status = succeeded6910worker2 # [ 148.279285] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6911client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6912client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6913client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6914worker2 # [ 149.092848] nix-daemon[851]: reaped child process 1666, status = succeeded6915client # farm-top> LOG-farm-top-during-drain6916worker2 # [ 149.169426] nix-daemon[851]: accepted connection from pid 1682, user nix-grpc-daemon (trusted)6917worker2 # [ 149.182970] nix-daemon[851]: reaped child process 1685, status = succeeded6918lb # [ 149.202913] niks3-server[1118]: 2026/09/23 05:14:19 INFO Received uploads request method=POST path=/api/pending_closures6919worker2 # [ 149.199979] nix-grpc-daemon[1525]: time=2026-09-23T05:14:19.365Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6920worker2 # [ 149.202055] nix-grpc-daemon[1525]: time=2026-09-23T05:14:19.366Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6921lb # [ 149.238957] niks3-server[1118]: 2026/09/23 05:14:19 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6922lb # [ 149.252233] niks3-server[1118]: 2026/09/23 05:14:19 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6923lb # [ 149.263433] niks3-server[1118]: 2026/09/23 05:14:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6924worker2 # [ 149.255929] nix-grpc-daemon[1525]: time=2026-09-23T05:14:19.421Z level=INFO msg="Uploading 1 narinfos"6925lb # [ 149.267577] niks3-server[1118]: 2026/09/23 05:14:19 INFO Signed narinfos id=60 count=16926lb # [ 149.273279] niks3-server[1118]: 2026/09/23 05:14:19 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6927lb # [ 149.283505] niks3-server[1118]: 2026/09/23 05:14:19 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6928lb # [ 149.287168] niks3-server[1118]: 2026/09/23 05:14:19 INFO Completed upload id=606929worker2 # [ 149.275808] nix-grpc-daemon[1525]: time=2026-09-23T05:14:19.441Z level=INFO msg="Upload complete. (190ms)"6930lb # [ 149.289697] niks3-server[1118]: 2026/09/23 05:14:19 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6931worker2 # [ 149.282076] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=16932worker2 # [ 149.296541] nix-daemon[851]: reaped child process 1664, status = succeeded6933client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&2, in 4.65 seconds)6934client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath6935client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath, in 0.11 seconds)6936worker1: must succeed: pkill -f 'read -t [9]0 x'6937worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.04 seconds)6938client: waiting for success: ! systemctl is-active sw6939worker1 # [ 149.638520] nix-daemon[864]: reaped child process 1861, status = succeeded6940worker1 # [ 149.724356] nix-daemon[864]: accepted connection from pid 1909, user nix-grpc-daemon (trusted)6941worker1 # [ 149.739642] nix-daemon[864]: reaped child process 1912, status = succeeded6942lb # [ 149.742798] niks3-server[1118]: 2026/09/23 05:14:19 INFO Received uploads request method=POST path=/api/pending_closures6943worker1 # [ 149.753561] nix-grpc-daemon[1798]: time=2026-09-23T05:14:19.746Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6944worker1 # [ 149.755174] nix-grpc-daemon[1798]: time=2026-09-23T05:14:19.748Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6945lb # [ 149.780971] niks3-server[1118]: 2026/09/23 05:14:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6946lb # [ 149.792264] niks3-server[1118]: 2026/09/23 05:14:19 INFO Signed narinfos id=61 count=16947worker1 # [ 149.797816] nix-grpc-daemon[1798]: time=2026-09-23T05:14:19.791Z level=INFO msg="Uploading 1 narinfos"6948lb # [ 149.798192] niks3-server[1118]: 2026/09/23 05:14:19 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6949lb # [ 149.804670] niks3-server[1118]: 2026/09/23 05:14:19 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6950lb # [ 149.809950] niks3-server[1118]: 2026/09/23 05:14:19 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6951lb # [ 149.819209] niks3-server[1118]: 2026/09/23 05:14:19 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6952lb # [ 149.823301] niks3-server[1118]: 2026/09/23 05:14:19 INFO Completed upload id=616953worker1 # [ 149.825939] nix-grpc-daemon[1798]: time=2026-09-23T05:14:19.819Z level=INFO msg="Upload complete. (216ms)"6954lb # [ 149.826108] niks3-server[1118]: 2026/09/23 05:14:19 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6955worker1 # [ 149.833808] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42172 duration_s=7 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=16956worker1 # [ 149.848702] nix-daemon[864]: reaped child process 1859, status = succeeded6957client # [ 149.866710] systemd[1]: sw.service: Deactivated successfully.6958client # [ 149.871264] systemd[1]: sw.service: Consumed 118ms CPU time over 8.763s wall clock time, 41.4M memory peak, 7.4K incoming IP traffic, 8.4K outgoing IP traffic.6959worker1 # [ 150.222975] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:20Z level=info event=signal_exit6960lb # [ 150.230827] niks3-server[1118]: 2026/09/23 05:14:20 INFO lead: released remote=192.168.1.3:491526961client: (finished: waiting for success: ! systemctl is-active sw, in 1.06 seconds)6962client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6963client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.03 seconds)6964worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6965worker2 # [ 151.219488] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:21Z level=info event=scheduler_restarting addr=lb:500516966worker1 # [ 151.235695] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:21Z level=info event=scheduler_restarting6967worker1 # [ 151.239603] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:21Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:421666968worker1 # [ 151.245527] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:21Z level=info event=worker_gone addr=192.168.1.4:500516969worker1 # [ 151.249244] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:21Z level=info event=scheduler_restarting addr=lb:500516970worker1 # [ 151.252806] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:21Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:421666971worker1 # [ 151.256570] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:21Z level=info event=worker_gone addr=192.168.1.3:500516972worker1 # [ 151.272627] nix-daemon[864]: reaped child process 1769, status = succeeded6973worker2 # [ 151.419940] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:21Z level=info event=scheduler_connected addr=lb:500516974worker2 # [ 151.424184] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6975worker1 # [ 151.451669] nix-grpc-daemon[1714]: ts=2026-09-23T05:14:21Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6976worker2 # [ 151.625321] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:21Z level=info event=scheduler_connected addr=lb:500516977worker2 # [ 151.629330] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6978worker2 # [ 151.830741] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:21Z level=info event=scheduler_connected addr=lb:500516979worker2 # [ 151.834690] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6980worker1 # [ 151.925444] nix-grpc-daemon[1714]: E0000 00:00:1790140461.918746 1760 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 20, started true, shutdown true6981worker1 # [ 151.931077] nix-grpc-daemon[1714]: E0000 00:00:1790140461.924068 1760 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6982worker2 # [ 152.035590] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:22Z level=info event=scheduler_connected addr=lb:500516983worker2 # [ 152.039556] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6984worker2 # [ 152.241769] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:22Z level=info event=scheduler_connected addr=lb:500516985worker2 # [ 152.245222] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6986worker2 # [ 152.446530] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:22Z level=info event=scheduler_connected addr=lb:500516987worker2 # [ 152.450543] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6988worker2 # [ 152.651579] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:22Z level=info event=scheduler_connected addr=lb:500516989worker2 # [ 152.655552] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6990worker2 # [ 152.856691] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:23Z level=info event=scheduler_connected addr=lb:500516991worker2 # [ 152.860630] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6992worker2 # [ 153.062074] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:23Z level=info event=scheduler_connected addr=lb:500516993worker2 # [ 153.066153] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6994worker1 # [ 153.242731] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6995worker1 # [ 153.246479] systemd[1]: nix-grpc-daemon.service: Consumed 583ms CPU time over 22.149s wall clock time, 36.1M memory peak, 24.9K incoming IP traffic, 32.6K outgoing IP traffic.6996worker1 # [ 153.263167] systemd[1]: Starting Nix worker-protocol over gRPC...6997worker2 # [ 153.268374] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:23Z level=info event=scheduler_connected addr=lb:500516998worker2 # [ 153.271571] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6999lb # [ 153.325383] niks3-server[1118]: 2026/09/23 05:14:23 INFO lead: acquired remote=192.168.1.4:605567000worker2 # [ 153.314067] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:23Z level=info event=scheduler_take_over7001worker1 # [ 153.358214] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:23Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"7002worker1 # [ 153.362924] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:23Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc7003worker1 # [ 153.366303] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:23Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=17004worker1 # [ 153.368264] nix-grpc-daemon[1926]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR7005worker1 # [ 153.369549] nix-grpc-daemon[1926]: E0000 00:00:1790140463.359280 1926 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.7006worker1 # [ 153.378419] nix-grpc-daemon[1926]: E0000 00:00:1790140463.366733 1926 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.7007worker1 # [ 153.380611] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:23Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket7008worker1 # [ 153.386145] systemd[1]: Started Nix worker-protocol over gRPC.7009worker1 # [ 153.389107] nix-grpc-daemon[1926]: E0000 00:00:1790140463.377309 1968 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.7010worker1 # [ 153.398702] nix-grpc-daemon[1926]: E0000 00:00:1790140463.392282 1968 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.7011worker1 # [ 153.402169] nix-grpc-daemon[1926]: E0000 00:00:1790140463.394573 1968 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.7012worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.84 seconds)7013lb: waiting until exactly no endpoints of x86_64-linux are unhealthy7014lb: must succeed: curl -sf localhost:9901/clusters7015worker1 # [ 153.414514] nix-grpc-daemon[1926]: E0000 00:00:1790140463.408029 1968 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.7016worker1 # [ 153.425984] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:23Z level=info event=scheduler_connected addr=lb:500517017worker1 # [ 153.427627] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7018lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7019worker2 # [ 153.469802] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:23Z level=info event=scheduler_connected addr=lb:500517020worker2 # [ 153.473734] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7021worker1 # [ 153.628933] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:23Z level=info event=scheduler_connected addr=lb:500517022worker1 # [ 153.633132] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7023worker2 # [ 153.675071] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:23Z level=info event=scheduler_connected addr=lb:500517024worker2 # [ 153.679133] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7025worker1 # [ 153.834198] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:23Z level=info event=scheduler_connected addr=lb:500517026worker1 # [ 153.838324] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7027worker2 # [ 153.880424] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:24Z level=info event=scheduler_connected addr=lb:500517028worker2 # [ 153.884479] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7029worker1 # [ 154.039336] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:24Z level=info event=scheduler_connected addr=lb:500517030worker1 # [ 154.043686] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7031worker2 # [ 154.085643] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:24Z level=info event=scheduler_connected addr=lb:500517032worker2 # [ 154.089703] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7033worker1 # [ 154.245429] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:24Z level=info event=scheduler_connected addr=lb:500517034worker2 # [ 154.240074] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:24Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:432527035worker2 # [ 154.244922] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:24Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=07036worker2 # [ 154.292712] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:24Z level=info event=scheduler_connected addr=lb:500517037worker2 # [ 154.301158] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:24Z level=info event=oidc_jwks provider=mock keys=17038worker2 # [ 154.305376] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:24Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:432527039worker2 # [ 154.309445] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:24Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=07040lb: must succeed: curl -sf localhost:9901/clusters7041lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7042lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.10 seconds)7043lb: must succeed: curl -sf localhost:9901/clusters7044lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7045worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7046worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7047(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 13.52 seconds)7048subtest: queue gauges show a burst while every build blocks7049lb: must succeed: curl -sf localhost:9901/clusters7050lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)7051lb: must succeed: curl -sf localhost:9901/clusters7052lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7053worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7054worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7055client: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]'7056client # [ 154.741508] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto --impure --expr "map (tag: import /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix { inherit tag; }) [ \"q0\" \"q1\" \"q2\" \"q3\" ]".7057client # Running as unit: burst.service; invocation ID: 158dbd0397774812b9bbefb9acd499617058client: (finished: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]', in 0.06 seconds)7059lb: must succeed: curl -sf localhost:9901/clusters7060lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7061lb: must succeed: curl -sf localhost:9901/clusters7062lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7063worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7064worker2 # [ 154.866450] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:25Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:374507065lb # [ 154.881143] envoy[903]: conn peer=[2001:db8:1::1]:38002 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9624 tx=4398 ms=44857066client # [ 154.885118] nix[1504]: don't know how to build these paths:7067client # [ 154.886030] nix[1504]: /nix/store/2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv7068lb # [ 154.884815] envoy[903]: conn peer=[2001:db8:1::1]:55298 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2975 tx=1541 ms=86237069client # [ 154.886953] nix[1504]: /nix/store/885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv7070worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7071client # [ 154.888051] nix[1504]: /nix/store/8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv7072client # [ 154.889162] nix[1504]: /nix/store/qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv7073lb # [ 154.889334] envoy[903]: conn peer=[2001:db8:1::3]:52008 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=975 tx=361 ms=201887074worker2 # [ 154.886112] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:25Z level=debug event=want client=1 drv=2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv cached=07075worker2 # [ 154.890653] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:25Z level=debug event=assigned drv=2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv worker=192.168.1.3:50051 clients=17076worker2 # [ 154.894349] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:25Z level=debug event=want client=1 drv=885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv cached=07077worker2 # [ 154.897461] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:25Z level=debug event=assigned drv=885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv worker=192.168.1.4:50051 clients=17078worker2 # [ 154.900947] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:25Z level=debug event=want client=1 drv=8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv cached=07079worker2 # [ 154.904171] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:25Z level=debug event=want client=1 drv=qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv cached=07080worker2 # [ 154.930533] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374507081worker1 # [ 154.948814] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567082worker2 # [ 154.942867] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374507083worker1 # [ 154.960829] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7084client # [ 154.964257] nix[1504]: copying 1 paths to 'lb:50051'...7085worker2 # [ 154.954171] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374507086client # [ 154.967173] nix[1504]: copying path '/nix/store/885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv' to 'lb:50051'...7087worker1 # [ 154.977542] nix-grpc-daemon[1926]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 45 ms (attempt 1/5)7088worker2 # [ 154.967249] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7089worker1 # [ 155.030178] nix-grpc-daemon[1926]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 78 ms (attempt 2/5)7090worker2 # [ 155.060571] nix-daemon[851]: accepted connection from pid 1700, user nix-grpc-daemon (trusted)7091worker2 # [ 155.073213] nix-daemon[851]: reaped child process 1703, status = succeeded7092lb # [ 155.093475] niks3-server[1118]: 2026/09/23 05:14:25 INFO Received uploads request method=POST path=/api/pending_closures7093worker2 # [ 155.090929] nix-grpc-daemon[1525]: time=2026-09-23T05:14:25.256Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7094worker2 # [ 155.092875] nix-grpc-daemon[1525]: time=2026-09-23T05:14:25.257Z level=INFO msg="Uploading 885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv (416B)"7095worker1 # [ 155.110368] nix-grpc-daemon[1926]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 215 ms (attempt 3/5)7096lb # [ 155.126381] niks3-server[1118]: 2026/09/23 05:14:25 INFO Registered completed upload object_key=885n55y9srkrjs4p487lr255wvhla7hq.ls7097lb # [ 155.139490] niks3-server[1118]: 2026/09/23 05:14:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign7098worker2 # [ 155.135771] nix-grpc-daemon[1525]: time=2026-09-23T05:14:25.300Z level=INFO msg="Uploading 1 narinfos"7099lb # [ 155.147114] niks3-server[1118]: 2026/09/23 05:14:25 INFO Signed narinfos id=62 count=17100lb # [ 155.159907] niks3-server[1118]: 2026/09/23 05:14:25 INFO Registered completed upload object_key=nar/0mmcg3cvna05ngbx6v5vspkmc5gj8lna2zc48rllhxr0g8i9mg6y.nar.zst7101lb # [ 155.170212] niks3-server[1118]: 2026/09/23 05:14:25 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete7102lb # [ 155.173872] niks3-server[1118]: 2026/09/23 05:14:25 INFO Completed upload id=627103lb # [ 155.175977] niks3-server[1118]: 2026/09/23 05:14:25 INFO Registered completed upload object_key=885n55y9srkrjs4p487lr255wvhla7hq.narinfo7104worker2 # [ 155.163718] nix-grpc-daemon[1525]: time=2026-09-23T05:14:25.328Z level=INFO msg="Upload complete. (187ms)"7105worker2 # [ 155.168734] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=0 paths=1 nar_bytes_in=4167106worker2 # [ 155.175093] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374507107worker2 # [ 155.188817] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7108worker2 # [ 155.197668] nix-daemon[851]: reaped child process 1698, status = succeeded7109worker2 # [ 155.202545] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7110client # [ 155.213306] nix[1504]: building '/nix/store/885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv' on lb:50051...7111worker1 # [ 155.327729] nix-grpc-daemon[1926]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 784 ms (attempt 4/5)7112lb: must succeed: curl -sf localhost:9901/clusters7113lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7114lb: must succeed: curl -sf localhost:9901/clusters7115lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7116worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7117worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)7118worker1 # [ 156.114816] nix-grpc-daemon[1926]: disabling binary cache 'https://cache.nixos.org' for 60 seconds7119worker1 # [ 156.118670] nix-grpc-daemon[1926]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org7120worker1 # [ 156.137395] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608567121client # [ 156.146827] nix[1504]: copying 1 paths to 'lb:50051'...7122client # [ 156.150587] nix[1504]: copying path '/nix/store/2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv' to 'lb:50051'...7123worker1 # [ 156.153675] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608567124worker1 # [ 156.167070] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7125worker1 # [ 156.188487] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:26Z level=info event=niks3_push_started pid=19917126worker1 # [ 156.298026] nix-grpc-daemon[1991]: time=2026-09-23T05:14:26.291Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""7127worker1 # [ 156.393620] nix-daemon[864]: accepted connection from pid 2004, user nix-grpc-daemon (trusted)7128worker1 # [ 156.407290] nix-daemon[864]: reaped child process 2007, status = succeeded7129lb # [ 156.412165] niks3-server[1118]: 2026/09/23 05:14:26 INFO Received uploads request method=POST path=/api/pending_closures7130worker1 # [ 156.423531] nix-grpc-daemon[1991]: time=2026-09-23T05:14:26.417Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7131worker1 # [ 156.425136] nix-grpc-daemon[1991]: time=2026-09-23T05:14:26.418Z level=INFO msg="Uploading 2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv (416B)"7132lb # [ 156.448144] niks3-server[1118]: 2026/09/23 05:14:26 INFO Registered completed upload object_key=nar/0raymp7y3ngiv7lj2qq6nkp63sms2510yhhlnr7hnz7ldnl7b3qc.nar.zst7133lb # [ 156.455677] niks3-server[1118]: 2026/09/23 05:14:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign7134lb # [ 156.459862] niks3-server[1118]: 2026/09/23 05:14:26 INFO Signed narinfos id=63 count=17135worker1 # [ 156.464591] nix-grpc-daemon[1991]: time=2026-09-23T05:14:26.458Z level=INFO msg="Uploading 1 narinfos"7136lb # [ 156.469886] niks3-server[1118]: 2026/09/23 05:14:26 INFO Registered completed upload object_key=2s9psgqm7awsyhlfyaclma7yr7qmrk0g.ls7137lb # [ 156.475059] niks3-server[1118]: 2026/09/23 05:14:26 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete7138lb # [ 156.478794] niks3-server[1118]: 2026/09/23 05:14:26 INFO Completed upload id=637139worker1 # [ 156.482514] nix-grpc-daemon[1991]: time=2026-09-23T05:14:26.476Z level=INFO msg="Upload complete. (181ms)"7140lb # [ 156.482460] niks3-server[1118]: 2026/09/23 05:14:26 INFO Registered completed upload object_key=2s9psgqm7awsyhlfyaclma7yr7qmrk0g.narinfo7141worker1 # [ 156.484860] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 paths=1 nar_bytes_in=4167142worker1 # [ 156.489059] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567143worker1 # [ 156.495268] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7144worker1 # [ 156.498374] nix-daemon[864]: reaped child process 1989, status = succeeded7145worker1 # [ 156.504813] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7146client # [ 156.500244] nix[1504]: building '/nix/store/2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv' on lb:50051...7147worker2 # [ 157.447508] nix-daemon[851]: reaped child process 1707, status = succeeded7148worker2 # [ 157.521418] nix-daemon[851]: accepted connection from pid 1735, user nix-grpc-daemon (trusted)7149worker2 # [ 157.533870] nix-daemon[851]: reaped child process 1738, status = succeeded7150lb # [ 157.554213] niks3-server[1118]: 2026/09/23 05:14:27 INFO Received uploads request method=POST path=/api/pending_closures7151worker2 # [ 157.550518] nix-grpc-daemon[1525]: time=2026-09-23T05:14:27.715Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7152worker2 # [ 157.552476] nix-grpc-daemon[1525]: time=2026-09-23T05:14:27.717Z level=INFO msg="Uploading nv0kp5b571hsjibnd9v06981i024iima-slow-q2 (120B)"7153lb # [ 157.591441] niks3-server[1118]: 2026/09/23 05:14:27 INFO Registered completed upload object_key=log/885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv7154lb # [ 157.604708] niks3-server[1118]: 2026/09/23 05:14:27 INFO Registered completed upload object_key=nar/15imdx7zyy6mzjjv6qvdzywh0d9w1n6ikc0rq4rq95aw9x6swfzj.nar.zst7155lb # [ 157.612412] niks3-server[1118]: 2026/09/23 05:14:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign7156lb # [ 157.618137] niks3-server[1118]: 2026/09/23 05:14:27 INFO Signed narinfos id=64 count=17157worker2 # [ 157.606362] nix-grpc-daemon[1525]: time=2026-09-23T05:14:27.771Z level=INFO msg="Uploading 1 narinfos"7158lb # [ 157.629226] niks3-server[1118]: 2026/09/23 05:14:27 INFO Registered completed upload object_key=nv0kp5b571hsjibnd9v06981i024iima.ls7159lb # [ 157.637022] niks3-server[1118]: 2026/09/23 05:14:27 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete7160lb # [ 157.639784] niks3-server[1118]: 2026/09/23 05:14:27 INFO Completed upload id=647161worker2 # [ 157.628451] nix-grpc-daemon[1525]: time=2026-09-23T05:14:27.793Z level=INFO msg="Upload complete. (217ms)"7162lb # [ 157.642568] niks3-server[1118]: 2026/09/23 05:14:27 INFO Registered completed upload object_key=nv0kp5b571hsjibnd9v06981i024iima.narinfo7163worker2 # [ 157.635625] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=2 drv=885n55y9srkrjs4p487lr255wvhla7hq-slow-q2.drv assign_id=2 outputs=17164worker2 # [ 157.645399] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:27Z level=debug event=assigned drv=8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv worker=192.168.1.4:50051 clients=17165worker2 # [ 157.655760] nix-daemon[851]: reaped child process 1705, status = succeeded7166worker2 # [ 157.687992] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374507167worker2 # [ 157.700416] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374507168client # [ 157.722773] nix[1504]: copying 1 paths to 'lb:50051'...7169worker2 # [ 157.711737] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374507170client # [ 157.724723] nix[1504]: copying path '/nix/store/8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv' to 'lb:50051'...7171worker2 # [ 157.724178] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7172worker2 # [ 157.819671] nix-daemon[851]: accepted connection from pid 1742, user nix-grpc-daemon (trusted)7173worker2 # [ 157.831791] nix-daemon[851]: reaped child process 1745, status = succeeded7174lb # [ 157.851985] niks3-server[1118]: 2026/09/23 05:14:27 INFO Received uploads request method=POST path=/api/pending_closures7175worker2 # [ 157.848402] nix-grpc-daemon[1525]: time=2026-09-23T05:14:28.013Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7176worker2 # [ 157.849919] nix-grpc-daemon[1525]: time=2026-09-23T05:14:28.015Z level=INFO msg="Uploading 8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv (416B)"7177lb # [ 157.882804] niks3-server[1118]: 2026/09/23 05:14:27 INFO Registered completed upload object_key=nar/1xibay372y9by9a678y9h7bfzwahsxi0vq51acgywak31q6q2m87.nar.zst7178lb # [ 157.891341] niks3-server[1118]: 2026/09/23 05:14:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7179lb # [ 157.896890] niks3-server[1118]: 2026/09/23 05:14:27 INFO Signed narinfos id=65 count=17180worker2 # [ 157.887975] nix-grpc-daemon[1525]: time=2026-09-23T05:14:28.053Z level=INFO msg="Uploading 1 narinfos"7181lb # [ 157.910773] niks3-server[1118]: 2026/09/23 05:14:27 INFO Registered completed upload object_key=8v867csm7d9sip4xd4nlg9jwvrpqqwv9.ls7182lb # [ 157.918370] niks3-server[1118]: 2026/09/23 05:14:27 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7183lb # [ 157.926663] niks3-server[1118]: 2026/09/23 05:14:27 INFO Completed upload id=657184worker2 # [ 157.915422] nix-grpc-daemon[1525]: time=2026-09-23T05:14:28.080Z level=INFO msg="Upload complete. (180ms)"7185lb # [ 157.930106] niks3-server[1118]: 2026/09/23 05:14:27 INFO Registered completed upload object_key=8v867csm7d9sip4xd4nlg9jwvrpqqwv9.narinfo7186worker2 # [ 157.919920] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=0 paths=1 nar_bytes_in=4167187worker2 # [ 157.925988] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374507188worker2 # [ 157.935694] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7189worker2 # [ 157.944946] nix-daemon[851]: reaped child process 1740, status = succeeded7190worker2 # [ 157.950752] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7191client # [ 157.960562] nix[1504]: building '/nix/store/8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv' on lb:50051...7192worker1 # [ 158.542604] nix-daemon[864]: reaped child process 2011, status = succeeded7193worker1 # [ 158.633096] nix-daemon[864]: accepted connection from pid 2036, user nix-grpc-daemon (trusted)7194worker1 # [ 158.645609] nix-daemon[864]: reaped child process 2039, status = succeeded7195lb # [ 158.652383] niks3-server[1118]: 2026/09/23 05:14:28 INFO Received uploads request method=POST path=/api/pending_closures7196worker1 # [ 158.664234] nix-grpc-daemon[1991]: time=2026-09-23T05:14:28.657Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7197worker1 # [ 158.666983] nix-grpc-daemon[1991]: time=2026-09-23T05:14:28.659Z level=INFO msg="Uploading vn3s5fgnxq5d2l4d4nn96l2q9wfdg4mq-slow-q3 (120B)"7198lb # [ 158.691653] niks3-server[1118]: 2026/09/23 05:14:28 INFO Registered completed upload object_key=log/2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv7199lb # [ 158.706463] niks3-server[1118]: 2026/09/23 05:14:28 INFO Registered completed upload object_key=nar/1v1grzkvw3yj09x0dkdm2avihvqvnri156qzp768hk85zbw7lkn9.nar.zst7200lb # [ 158.711389] niks3-server[1118]: 2026/09/23 05:14:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7201lb # [ 158.716215] niks3-server[1118]: 2026/09/23 05:14:28 INFO Signed narinfos id=66 count=17202worker1 # [ 158.718491] nix-grpc-daemon[1991]: time=2026-09-23T05:14:28.712Z level=INFO msg="Uploading 1 narinfos"7203lb # [ 158.723832] niks3-server[1118]: 2026/09/23 05:14:28 INFO Registered completed upload object_key=vn3s5fgnxq5d2l4d4nn96l2q9wfdg4mq.ls7204lb # [ 158.732500] niks3-server[1118]: 2026/09/23 05:14:28 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7205lb # [ 158.736888] niks3-server[1118]: 2026/09/23 05:14:28 INFO Completed upload id=667206worker1 # [ 158.739516] nix-grpc-daemon[1991]: time=2026-09-23T05:14:28.733Z level=INFO msg="Upload complete. (226ms)"7207lb # [ 158.740133] niks3-server[1118]: 2026/09/23 05:14:28 INFO Registered completed upload object_key=vn3s5fgnxq5d2l4d4nn96l2q9wfdg4mq.narinfo7208worker2 # [ 158.731984] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:28Z level=debug event=assigned drv=qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv worker=192.168.1.3:50051 clients=17209worker1 # [ 158.747495] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=2 drv=2s9psgqm7awsyhlfyaclma7yr7qmrk0g-slow-q3.drv assign_id=1 outputs=17210worker1 # [ 158.762141] nix-daemon[864]: reaped child process 2009, status = succeeded7211worker1 # [ 158.791386] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567212worker1 # [ 158.804430] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608567213client # [ 158.812789] nix[1504]: copying 1 paths to 'lb:50051'...7214worker1 # [ 158.815971] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608567215client # [ 158.814728] nix[1504]: copying path '/nix/store/qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv' to 'lb:50051'...7216worker1 # [ 158.828172] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7217worker1 # [ 158.931790] nix-daemon[864]: accepted connection from pid 2043, user nix-grpc-daemon (trusted)7218worker1 # [ 158.945740] nix-daemon[864]: reaped child process 2046, status = succeeded7219lb # [ 158.951081] niks3-server[1118]: 2026/09/23 05:14:28 INFO Received uploads request method=POST path=/api/pending_closures7220worker1 # [ 158.961536] nix-grpc-daemon[1991]: time=2026-09-23T05:14:28.955Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7221worker1 # [ 158.963094] nix-grpc-daemon[1991]: time=2026-09-23T05:14:28.956Z level=INFO msg="Uploading qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv (416B)"7222lb # [ 158.982388] niks3-server[1118]: 2026/09/23 05:14:29 INFO Registered completed upload object_key=nar/1spxkiwzvlj9sgbjal4jf4p68h5kbylxymfww73s5va157qxgafi.nar.zst7223lb # [ 158.989413] niks3-server[1118]: 2026/09/23 05:14:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7224lb # [ 158.995916] niks3-server[1118]: 2026/09/23 05:14:29 INFO Signed narinfos id=67 count=17225worker1 # [ 159.002463] nix-grpc-daemon[1991]: time=2026-09-23T05:14:28.996Z level=INFO msg="Uploading 1 narinfos"7226lb # [ 159.005135] niks3-server[1118]: 2026/09/23 05:14:29 INFO Registered completed upload object_key=qs0cgdr560r5cllf2djj9bb4gz0vrp4p.ls7227lb # [ 159.015301] niks3-server[1118]: 2026/09/23 05:14:29 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7228lb # [ 159.021329] niks3-server[1118]: 2026/09/23 05:14:29 INFO Completed upload id=677229worker1 # [ 159.024166] nix-grpc-daemon[1991]: time=2026-09-23T05:14:29.017Z level=INFO msg="Upload complete. (184ms)"7230lb # [ 159.026588] niks3-server[1118]: 2026/09/23 05:14:29 INFO Registered completed upload object_key=qs0cgdr560r5cllf2djj9bb4gz0vrp4p.narinfo7231worker1 # [ 159.028710] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 paths=1 nar_bytes_in=4167232worker1 # [ 159.033795] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567233worker1 # [ 159.046065] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7234worker1 # [ 159.049728] nix-daemon[864]: reaped child process 2041, status = succeeded7235worker1 # [ 159.054798] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7236client # [ 159.051479] nix[1504]: building '/nix/store/qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv' on lb:50051...7237worker2 # [ 159.566331] nix-daemon[851]: reaped child process 1749, status = succeeded7238worker2 # [ 159.642091] nix-daemon[851]: accepted connection from pid 1774, user nix-grpc-daemon (trusted)7239worker2 # [ 159.653494] nix-daemon[851]: reaped child process 1777, status = succeeded7240lb # [ 159.673888] niks3-server[1118]: 2026/09/23 05:14:29 INFO Received uploads request method=POST path=/api/pending_closures7241worker2 # [ 159.670858] nix-grpc-daemon[1525]: time=2026-09-23T05:14:29.835Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7242worker2 # [ 159.672586] nix-grpc-daemon[1525]: time=2026-09-23T05:14:29.837Z level=INFO msg="Uploading wmw8972gq2nf739xhg7is7lylxf8299y-slow-q0 (120B)"7243lb # [ 159.722079] niks3-server[1118]: 2026/09/23 05:14:29 INFO Registered completed upload object_key=wmw8972gq2nf739xhg7is7lylxf8299y.ls7244lb # [ 159.725685] niks3-server[1118]: 2026/09/23 05:14:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7245lb # [ 159.735591] niks3-server[1118]: 2026/09/23 05:14:29 INFO Registered completed upload object_key=log/8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv7246lb # [ 159.747369] niks3-server[1118]: 2026/09/23 05:14:29 INFO Registered completed upload object_key=nar/0xr809jj32p5n6mnc6i7lqsz48wrm5f9jp14m77k724041711iq7.nar.zst7247lb # [ 159.752222] niks3-server[1118]: 2026/09/23 05:14:29 INFO Signed narinfos id=68 count=17248worker2 # [ 159.743565] nix-grpc-daemon[1525]: time=2026-09-23T05:14:29.908Z level=INFO msg="Uploading 1 narinfos"7249lb # [ 159.759510] niks3-server[1118]: 2026/09/23 05:14:29 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7250lb # [ 159.768210] niks3-server[1118]: 2026/09/23 05:14:29 INFO Registered completed upload object_key=wmw8972gq2nf739xhg7is7lylxf8299y.narinfo7251lb # [ 159.771702] niks3-server[1118]: 2026/09/23 05:14:29 INFO Completed upload id=687252worker2 # [ 159.760306] nix-grpc-daemon[1525]: time=2026-09-23T05:14:29.925Z level=INFO msg="Upload complete. (203ms)"7253worker2 # [ 159.767265] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=1 drv=8v867csm7d9sip4xd4nlg9jwvrpqqwv9-slow-q0.drv assign_id=3 outputs=17254worker2 # [ 159.782067] nix-daemon[851]: reaped child process 1747, status = succeeded7255worker1 # [ 160.671480] nix-daemon[864]: reaped child process 2050, status = succeeded7256worker1 # [ 160.751775] nix-daemon[864]: accepted connection from pid 2075, user nix-grpc-daemon (trusted)7257worker1 # [ 160.764332] nix-daemon[864]: reaped child process 2078, status = succeeded7258lb # [ 160.769596] niks3-server[1118]: 2026/09/23 05:14:30 INFO Received uploads request method=POST path=/api/pending_closures7259worker1 # [ 160.782884] nix-grpc-daemon[1991]: time=2026-09-23T05:14:30.776Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7260worker1 # [ 160.784413] nix-grpc-daemon[1991]: time=2026-09-23T05:14:30.778Z level=INFO msg="Uploading 106i40yblbkazayvn9ppswrq6kqs26sj-slow-q1 (120B)"7261lb # [ 160.822058] niks3-server[1118]: 2026/09/23 05:14:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7262lb # [ 160.827096] niks3-server[1118]: 2026/09/23 05:14:30 INFO Registered completed upload object_key=log/qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv7263lb # [ 160.830831] niks3-server[1118]: 2026/09/23 05:14:30 INFO Registered completed upload object_key=106i40yblbkazayvn9ppswrq6kqs26sj.ls7264worker1 # [ 160.838295] nix-grpc-daemon[1991]: time=2026-09-23T05:14:30.831Z level=INFO msg="Uploading 1 narinfos"7265lb # [ 160.835964] niks3-server[1118]: 2026/09/23 05:14:30 INFO Signed narinfos id=69 count=17266lb # [ 160.839079] niks3-server[1118]: 2026/09/23 05:14:30 INFO Registered completed upload object_key=nar/0c1c367r3msa4fz6imwp70zxi4fxvrkb86v3hv921wsrf13yd8v3.nar.zst7267lb # [ 160.848266] niks3-server[1118]: 2026/09/23 05:14:30 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7268lb # [ 160.852173] niks3-server[1118]: 2026/09/23 05:14:30 INFO Completed upload id=697269worker1 # [ 160.854893] nix-grpc-daemon[1991]: time=2026-09-23T05:14:30.848Z level=INFO msg="Upload complete. (192ms)"7270lb # [ 160.854095] niks3-server[1118]: 2026/09/23 05:14:30 INFO Registered completed upload object_key=106i40yblbkazayvn9ppswrq6kqs26sj.narinfo7271worker1 # [ 160.860691] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=1 drv=qs0cgdr560r5cllf2djj9bb4gz0vrp4p-slow-q1.drv assign_id=4 outputs=17272worker1 # [ 160.874940] nix-daemon[864]: reaped child process 2048, status = succeeded7273client # [ 160.890515] systemd[1]: burst.service: Deactivated successfully.7274client # [ 160.894245] systemd[1]: burst.service: Consumed 160ms CPU time over 6.151s wall clock time, 53.9M memory peak, 14.2K incoming IP traffic, 19.2K outgoing IP traffic.7275client: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }7276client: (finished: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }, in 0.04 seconds)7277lb: must succeed: curl -sf localhost:9901/clusters7278lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7279lb: must succeed: curl -sf localhost:9901/clusters7280lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7281worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7282worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds)7283(finished: subtest: queue gauges show a burst while every build blocks, in 7.39 seconds)7284subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does7285client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&17286worker2 # [ 162.107106] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:374507287worker1 # [ 162.125856] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608567288worker2 # [ 162.152268] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:32Z level=debug event=want client=2 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=07289worker2 # [ 162.156353] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:32Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=17290worker2 # [ 162.160727] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:32Z level=debug event=want client=2 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=07291worker2 # [ 162.165180] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:32Z level=debug event=want client=2 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=07292worker1 # [ 162.223865] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567293worker1 # [ 162.238461] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608567294worker1 # [ 162.252437] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608567295worker1 # [ 162.266159] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7296worker1 # [ 162.371408] nix-daemon[864]: accepted connection from pid 2086, user nix-grpc-daemon (trusted)7297worker1 # [ 162.385929] nix-daemon[864]: reaped child process 2089, status = succeeded7298lb # [ 162.391884] niks3-server[1118]: 2026/09/23 05:14:32 INFO Received uploads request method=POST path=/api/pending_closures7299worker1 # [ 162.403635] nix-grpc-daemon[1991]: time=2026-09-23T05:14:32.396Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7300worker1 # [ 162.405102] nix-grpc-daemon[1991]: time=2026-09-23T05:14:32.396Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"7301lb # [ 162.420315] niks3-server[1118]: 2026/09/23 05:14:32 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst7302lb # [ 162.429752] niks3-server[1118]: 2026/09/23 05:14:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7303worker1 # [ 162.436140] nix-grpc-daemon[1991]: time=2026-09-23T05:14:32.429Z level=INFO msg="Uploading 1 narinfos"7304lb # [ 162.433491] niks3-server[1118]: 2026/09/23 05:14:32 INFO Signed narinfos id=70 count=17305lb # [ 162.438296] niks3-server[1118]: 2026/09/23 05:14:32 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls7306lb # [ 162.449834] niks3-server[1118]: 2026/09/23 05:14:32 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7307lb # [ 162.453537] niks3-server[1118]: 2026/09/23 05:14:32 INFO Completed upload id=707308worker1 # [ 162.456061] nix-grpc-daemon[1991]: time=2026-09-23T05:14:32.449Z level=INFO msg="Upload complete. (176ms)"7309lb # [ 162.456301] niks3-server[1118]: 2026/09/23 05:14:32 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo7310worker1 # [ 162.460186] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 paths=1 nar_bytes_in=5047311worker1 # [ 162.465330] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567312worker1 # [ 162.473261] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7313worker1 # [ 162.478945] nix-daemon[864]: reaped child process 2084, status = succeeded7314worker1 # [ 162.484073] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7315worker1 # [ 163.730538] nix-daemon[864]: reaped child process 2093, status = succeeded7316worker1 # [ 163.809187] nix-daemon[864]: accepted connection from pid 2109, user nix-grpc-daemon (trusted)7317worker1 # [ 163.822918] nix-daemon[864]: reaped child process 2112, status = succeeded7318lb # [ 163.829634] niks3-server[1118]: 2026/09/23 05:14:33 INFO Received uploads request method=POST path=/api/pending_closures7319worker1 # [ 163.840776] nix-grpc-daemon[1991]: time=2026-09-23T05:14:33.834Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7320worker1 # [ 163.842348] nix-grpc-daemon[1991]: time=2026-09-23T05:14:33.835Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"7321lb # [ 163.868900] niks3-server[1118]: 2026/09/23 05:14:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7322lb # [ 163.875368] niks3-server[1118]: 2026/09/23 05:14:33 INFO Signed narinfos id=71 count=17323worker1 # [ 163.882103] nix-grpc-daemon[1991]: time=2026-09-23T05:14:33.875Z level=INFO msg="Uploading 1 narinfos"7324lb # [ 163.885135] niks3-server[1118]: 2026/09/23 05:14:33 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv7325lb # [ 163.891619] niks3-server[1118]: 2026/09/23 05:14:33 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls7326lb # [ 163.897512] niks3-server[1118]: 2026/09/23 05:14:33 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst7327lb # [ 163.908216] niks3-server[1118]: 2026/09/23 05:14:33 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7328lb # [ 163.911324] niks3-server[1118]: 2026/09/23 05:14:33 INFO Completed upload id=717329lb # [ 163.913776] niks3-server[1118]: 2026/09/23 05:14:33 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo7330worker1 # [ 163.915217] nix-grpc-daemon[1991]: time=2026-09-23T05:14:33.908Z level=INFO msg="Upload complete. (224ms)"7331worker1 # [ 163.920857] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=5 outputs=17332worker2 # [ 163.908815] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:34Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=17333worker2 # [ 163.918238] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:34Z level=debug event=want client=2 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=07334worker1 # [ 163.936857] nix-daemon[864]: reaped child process 2091, status = succeeded7335worker1 # [ 163.966629] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567336worker1 # [ 163.980659] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608567337worker1 # [ 163.993747] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608567338worker1 # [ 164.003196] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7339worker1 # [ 164.102432] nix-daemon[864]: accepted connection from pid 2116, user nix-grpc-daemon (trusted)7340worker1 # [ 164.116126] nix-daemon[864]: reaped child process 2119, status = succeeded7341lb # [ 164.122307] niks3-server[1118]: 2026/09/23 05:14:34 INFO Received uploads request method=POST path=/api/pending_closures7342worker1 # [ 164.133392] nix-grpc-daemon[1991]: time=2026-09-23T05:14:34.126Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7343worker1 # [ 164.134885] nix-grpc-daemon[1991]: time=2026-09-23T05:14:34.128Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"7344lb # [ 164.154894] niks3-server[1118]: 2026/09/23 05:14:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7345lb # [ 164.161141] niks3-server[1118]: 2026/09/23 05:14:34 INFO Signed narinfos id=72 count=17346worker1 # [ 164.167282] nix-grpc-daemon[1991]: time=2026-09-23T05:14:34.160Z level=INFO msg="Uploading 1 narinfos"7347lb # [ 164.171877] niks3-server[1118]: 2026/09/23 05:14:34 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst7348lb # [ 164.177367] niks3-server[1118]: 2026/09/23 05:14:34 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls7349lb # [ 164.191553] niks3-server[1118]: 2026/09/23 05:14:34 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7350worker1 # [ 164.199635] nix-grpc-daemon[1991]: time=2026-09-23T05:14:34.193Z level=INFO msg="Upload complete. (184ms)"7351lb # [ 164.197686] niks3-server[1118]: 2026/09/23 05:14:34 INFO Completed upload id=727352worker1 # [ 164.202901] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 paths=1 nar_bytes_in=5047353lb # [ 164.202556] niks3-server[1118]: 2026/09/23 05:14:34 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo7354worker1 # [ 164.209428] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567355worker1 # [ 164.218433] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7356worker1 # [ 164.227725] nix-daemon[864]: reaped child process 2114, status = succeeded7357worker1 # [ 164.232871] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7358worker1 # [ 164.847521] nix-daemon[864]: reaped child process 2123, status = succeeded7359lb # [ 164.882403] envoy[903]: conn peer=[2001:db8:1::1]:58608 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9673 tx=5229 ms=59927360worker1 # [ 164.924279] nix-daemon[864]: accepted connection from pid 2139, user nix-grpc-daemon (trusted)7361worker1 # [ 164.938597] nix-daemon[864]: reaped child process 2142, status = succeeded7362lb # [ 164.946729] niks3-server[1118]: 2026/09/23 05:14:34 INFO Received uploads request method=POST path=/api/pending_closures7363worker1 # [ 164.956062] nix-grpc-daemon[1991]: time=2026-09-23T05:14:34.949Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7364worker1 # [ 164.957543] nix-grpc-daemon[1991]: time=2026-09-23T05:14:34.951Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"7365lb # [ 164.988637] niks3-server[1118]: 2026/09/23 05:14:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7366lb # [ 164.992193] niks3-server[1118]: 2026/09/23 05:14:35 INFO Signed narinfos id=73 count=17367worker1 # [ 164.997057] nix-grpc-daemon[1991]: time=2026-09-23T05:14:34.990Z level=INFO msg="Uploading 1 narinfos"7368lb # [ 164.999815] niks3-server[1118]: 2026/09/23 05:14:35 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst7369lb # [ 165.004870] niks3-server[1118]: 2026/09/23 05:14:35 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls7370lb # [ 165.014766] niks3-server[1118]: 2026/09/23 05:14:35 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7371lb # [ 165.017769] niks3-server[1118]: 2026/09/23 05:14:35 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv7372lb # [ 165.022117] niks3-server[1118]: 2026/09/23 05:14:35 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo7373lb # [ 165.024478] niks3-server[1118]: 2026/09/23 05:14:35 INFO Completed upload id=737374worker1 # [ 165.027138] nix-grpc-daemon[1991]: time=2026-09-23T05:14:35.020Z level=INFO msg="Upload complete. (189ms)"7375worker1 # [ 165.031993] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=6 outputs=17376worker2 # [ 165.019872] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:35Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=17377worker1 # [ 165.039402] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567378worker2 # [ 165.029133] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:35Z level=debug event=want client=2 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=07379worker1 # [ 165.050313] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608567380worker1 # [ 165.053961] nix-daemon[864]: reaped child process 2121, status = succeeded7381worker1 # [ 165.057527] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608567382worker1 # [ 165.061539] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7383worker1 # [ 165.150772] nix-daemon[864]: accepted connection from pid 2146, user nix-grpc-daemon (trusted)7384worker1 # [ 165.164476] nix-daemon[864]: reaped child process 2149, status = succeeded7385lb # [ 165.170167] niks3-server[1118]: 2026/09/23 05:14:35 INFO Received uploads request method=POST path=/api/pending_closures7386worker1 # [ 165.180564] nix-grpc-daemon[1991]: time=2026-09-23T05:14:35.174Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7387worker1 # [ 165.182688] nix-grpc-daemon[1991]: time=2026-09-23T05:14:35.175Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"7388lb # [ 165.200838] niks3-server[1118]: 2026/09/23 05:14:35 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst7389lb # [ 165.210490] niks3-server[1118]: 2026/09/23 05:14:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7390lb # [ 165.215587] niks3-server[1118]: 2026/09/23 05:14:35 INFO Signed narinfos id=74 count=17391worker1 # [ 165.223052] nix-grpc-daemon[1991]: time=2026-09-23T05:14:35.216Z level=INFO msg="Uploading 1 narinfos"7392lb # [ 165.223512] niks3-server[1118]: 2026/09/23 05:14:35 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls7393lb # [ 165.239502] niks3-server[1118]: 2026/09/23 05:14:35 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7394lb # [ 165.243284] niks3-server[1118]: 2026/09/23 05:14:35 INFO Completed upload id=747395worker1 # [ 165.246005] nix-grpc-daemon[1991]: time=2026-09-23T05:14:35.239Z level=INFO msg="Upload complete. (178ms)"7396lb # [ 165.245579] niks3-server[1118]: 2026/09/23 05:14:35 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo7397worker1 # [ 165.250646] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 paths=1 nar_bytes_in=5047398worker1 # [ 165.256232] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567399worker1 # [ 165.264843] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7400worker1 # [ 165.274472] nix-daemon[864]: reaped child process 2144, status = succeeded7401worker1 # [ 165.280563] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7402worker1 # [ 166.296465] nix-daemon[864]: reaped child process 2153, status = succeeded7403worker1 # [ 166.377539] nix-daemon[864]: accepted connection from pid 2169, user nix-grpc-daemon (trusted)7404worker1 # [ 166.392175] nix-daemon[864]: reaped child process 2172, status = succeeded7405lb # [ 166.396708] niks3-server[1118]: 2026/09/23 05:14:36 INFO Received uploads request method=POST path=/api/pending_closures7406worker1 # [ 166.407563] nix-grpc-daemon[1991]: time=2026-09-23T05:14:36.400Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7407worker1 # [ 166.410096] nix-grpc-daemon[1991]: time=2026-09-23T05:14:36.402Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"7408lb # [ 166.443261] niks3-server[1118]: 2026/09/23 05:14:36 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst7409lb # [ 166.448608] niks3-server[1118]: 2026/09/23 05:14:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7410lb # [ 166.454318] niks3-server[1118]: 2026/09/23 05:14:36 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv7411worker1 # [ 166.462681] nix-grpc-daemon[1991]: time=2026-09-23T05:14:36.456Z level=INFO msg="Uploading 1 narinfos"7412lb # [ 166.460151] niks3-server[1118]: 2026/09/23 05:14:36 INFO Signed narinfos id=75 count=17413lb # [ 166.472830] niks3-server[1118]: 2026/09/23 05:14:36 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7414lb # [ 166.479361] niks3-server[1118]: 2026/09/23 05:14:36 INFO Completed upload id=757415lb # [ 166.481887] niks3-server[1118]: 2026/09/23 05:14:36 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls7416worker1 # [ 166.483859] nix-grpc-daemon[1991]: time=2026-09-23T05:14:36.477Z level=INFO msg="Upload complete. (197ms)"7417lb # [ 166.485298] niks3-server[1118]: 2026/09/23 05:14:36 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo7418worker1 # [ 166.490863] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=7 outputs=17419worker2 # [ 166.477931] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:36Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=17420worker2 # [ 166.487717] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:36Z level=debug event=want client=2 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=07421worker1 # [ 166.505719] nix-daemon[864]: reaped child process 2151, status = succeeded7422worker1 # [ 166.542768] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567423worker1 # [ 166.555451] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608567424worker1 # [ 166.568872] nix-grpc-daemon[1991]: time=2026-09-23T05:14:36.562Z level=INFO msg="All 1 paths already cached"7425worker1 # [ 166.576652] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608567426worker1 # [ 166.585794] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7427worker1 # [ 166.687650] nix-daemon[864]: accepted connection from pid 2176, user nix-grpc-daemon (trusted)7428worker1 # [ 166.702370] nix-daemon[864]: reaped child process 2179, status = succeeded7429lb # [ 166.706482] niks3-server[1118]: 2026/09/23 05:14:36 INFO Received uploads request method=POST path=/api/pending_closures7430worker1 # [ 166.716471] nix-grpc-daemon[1991]: time=2026-09-23T05:14:36.710Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7431worker1 # [ 166.718087] nix-grpc-daemon[1991]: time=2026-09-23T05:14:36.711Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"7432lb # [ 166.734072] niks3-server[1118]: 2026/09/23 05:14:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign7433lb # [ 166.741719] niks3-server[1118]: 2026/09/23 05:14:36 INFO Signed narinfos id=76 count=17434worker1 # [ 166.747953] nix-grpc-daemon[1991]: time=2026-09-23T05:14:36.741Z level=INFO msg="Uploading 1 narinfos"7435lb # [ 166.749379] niks3-server[1118]: 2026/09/23 05:14:36 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst7436lb # [ 166.755314] niks3-server[1118]: 2026/09/23 05:14:36 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls7437lb # [ 166.769725] niks3-server[1118]: 2026/09/23 05:14:36 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete7438worker1 # [ 166.777100] nix-grpc-daemon[1991]: time=2026-09-23T05:14:36.770Z level=INFO msg="Upload complete. (176ms)"7439lb # [ 166.774890] niks3-server[1118]: 2026/09/23 05:14:36 INFO Completed upload id=767440worker1 # [ 166.780088] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 paths=1 nar_bytes_in=6247441lb # [ 166.778986] niks3-server[1118]: 2026/09/23 05:14:36 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo7442worker1 # [ 166.786198] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567443worker1 # [ 166.796493] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7444worker1 # [ 166.802191] nix-daemon[864]: reaped child process 2174, status = succeeded7445worker1 # [ 166.807145] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7446worker1 # [ 167.822744] nix-daemon[864]: reaped child process 2183, status = succeeded7447worker1 # [ 167.901455] nix-daemon[864]: accepted connection from pid 2199, user nix-grpc-daemon (trusted)7448worker1 # [ 167.916049] nix-daemon[864]: reaped child process 2202, status = succeeded7449lb # [ 167.921351] niks3-server[1118]: 2026/09/23 05:14:37 INFO Received uploads request method=POST path=/api/pending_closures7450worker1 # [ 167.934142] nix-grpc-daemon[1991]: time=2026-09-23T05:14:37.927Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7451worker1 # [ 167.936421] nix-grpc-daemon[1991]: time=2026-09-23T05:14:37.929Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"7452lb # [ 167.963995] niks3-server[1118]: 2026/09/23 05:14:37 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls7453lb # [ 167.969461] niks3-server[1118]: 2026/09/23 05:14:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign7454lb # [ 167.975160] niks3-server[1118]: 2026/09/23 05:14:38 INFO Signed narinfos id=77 count=17455worker1 # [ 167.981182] nix-grpc-daemon[1991]: time=2026-09-23T05:14:37.974Z level=INFO msg="Uploading 1 narinfos"7456lb # [ 167.983854] niks3-server[1118]: 2026/09/23 05:14:38 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv7457lb # [ 167.989968] niks3-server[1118]: 2026/09/23 05:14:38 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst7458lb # [ 167.999832] niks3-server[1118]: 2026/09/23 05:14:38 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete7459lb # [ 168.004619] niks3-server[1118]: 2026/09/23 05:14:38 INFO Completed upload id=777460worker1 # [ 168.008950] nix-grpc-daemon[1991]: time=2026-09-23T05:14:38.002Z level=INFO msg="Upload complete. (195ms)"7461lb # [ 168.008724] niks3-server[1118]: 2026/09/23 05:14:38 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo7462worker1 # [ 168.015869] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=8 outputs=17463worker2 # [ 168.002903] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:38Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=17464worker2 # [ 168.012252] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:38Z level=debug event=want client=2 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=07465worker1 # [ 168.031363] nix-daemon[864]: reaped child process 2181, status = succeeded7466worker1 # [ 168.067453] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567467worker1 # [ 168.079961] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608567468worker1 # [ 168.093632] nix-grpc-daemon[1991]: time=2026-09-23T05:14:38.087Z level=INFO msg="All 1 paths already cached"7469worker1 # [ 168.100858] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608567470worker1 # [ 168.111137] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7471worker1 # [ 168.211387] nix-daemon[864]: accepted connection from pid 2206, user nix-grpc-daemon (trusted)7472worker1 # [ 168.224847] nix-daemon[864]: reaped child process 2209, status = succeeded7473lb # [ 168.230834] niks3-server[1118]: 2026/09/23 05:14:38 INFO Received uploads request method=POST path=/api/pending_closures7474worker1 # [ 168.241377] nix-grpc-daemon[1991]: time=2026-09-23T05:14:38.234Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7475worker1 # [ 168.243040] nix-grpc-daemon[1991]: time=2026-09-23T05:14:38.236Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"7476lb # [ 168.259637] niks3-server[1118]: 2026/09/23 05:14:38 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls7477lb # [ 168.269289] niks3-server[1118]: 2026/09/23 05:14:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign7478lb # [ 168.276587] niks3-server[1118]: 2026/09/23 05:14:38 INFO Signed narinfos id=78 count=17479worker1 # [ 168.282762] nix-grpc-daemon[1991]: time=2026-09-23T05:14:38.276Z level=INFO msg="Uploading 1 narinfos"7480lb # [ 168.283837] niks3-server[1118]: 2026/09/23 05:14:38 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst7481lb # [ 168.296397] niks3-server[1118]: 2026/09/23 05:14:38 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete7482lb # [ 168.304210] niks3-server[1118]: 2026/09/23 05:14:38 INFO Completed upload id=787483worker1 # [ 168.306504] nix-grpc-daemon[1991]: time=2026-09-23T05:14:38.299Z level=INFO msg="Upload complete. (183ms)"7484lb # [ 168.308069] niks3-server[1118]: 2026/09/23 05:14:38 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo7485worker1 # [ 168.310627] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 paths=1 nar_bytes_in=6247486worker1 # [ 168.315420] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567487worker1 # [ 168.328073] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7488worker1 # [ 168.332169] nix-daemon[864]: reaped child process 2204, status = succeeded7489worker1 # [ 168.337263] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7490worker1 # [ 169.385198] nix-daemon[864]: reaped child process 2213, status = succeeded7491worker1 # [ 169.462924] nix-daemon[864]: accepted connection from pid 2229, user nix-grpc-daemon (trusted)7492worker1 # [ 169.476868] nix-daemon[864]: reaped child process 2232, status = succeeded7493lb # [ 169.482764] niks3-server[1118]: 2026/09/23 05:14:39 INFO Received uploads request method=POST path=/api/pending_closures7494worker1 # [ 169.494434] nix-grpc-daemon[1991]: time=2026-09-23T05:14:39.487Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7495worker1 # [ 169.495982] nix-grpc-daemon[1991]: time=2026-09-23T05:14:39.489Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"7496lb # [ 169.530606] niks3-server[1118]: 2026/09/23 05:14:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign7497worker1 # [ 169.541693] nix-grpc-daemon[1991]: time=2026-09-23T05:14:39.534Z level=INFO msg="Uploading 1 narinfos"7498lb # [ 169.540168] niks3-server[1118]: 2026/09/23 05:14:39 INFO Signed narinfos id=79 count=17499lb # [ 169.542792] niks3-server[1118]: 2026/09/23 05:14:39 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls7500lb # [ 169.549407] niks3-server[1118]: 2026/09/23 05:14:39 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv7501lb # [ 169.555078] niks3-server[1118]: 2026/09/23 05:14:39 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst7502lb # [ 169.566931] niks3-server[1118]: 2026/09/23 05:14:39 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete7503lb # [ 169.573498] niks3-server[1118]: 2026/09/23 05:14:39 INFO Completed upload id=797504worker1 # [ 169.576043] nix-grpc-daemon[1991]: time=2026-09-23T05:14:39.569Z level=INFO msg="Upload complete. (230ms)"7505lb # [ 169.576273] niks3-server[1118]: 2026/09/23 05:14:39 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo7506worker2 # [ 169.571478] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:39Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=17507worker1 # [ 169.584355] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=9 outputs=17508worker1 # [ 169.592391] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567509worker2 # [ 169.579349] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:39Z level=debug event=want client=2 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=07510worker1 # [ 169.602975] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608567511worker1 # [ 169.612683] nix-daemon[864]: reaped child process 2211, status = succeeded7512worker1 # [ 169.615744] nix-grpc-daemon[1991]: time=2026-09-23T05:14:39.609Z level=INFO msg="All 1 paths already cached"7513worker1 # [ 169.620394] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608567514worker1 # [ 169.624300] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7515worker1 # [ 169.716929] nix-daemon[864]: accepted connection from pid 2236, user nix-grpc-daemon (trusted)7516worker1 # [ 169.731222] nix-daemon[864]: reaped child process 2239, status = succeeded7517lb # [ 169.736294] niks3-server[1118]: 2026/09/23 05:14:39 INFO Received uploads request method=POST path=/api/pending_closures7518worker1 # [ 169.746496] nix-grpc-daemon[1991]: time=2026-09-23T05:14:39.740Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7519worker1 # [ 169.748539] nix-grpc-daemon[1991]: time=2026-09-23T05:14:39.741Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"7520lb # [ 169.766725] niks3-server[1118]: 2026/09/23 05:14:39 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst7521lb # [ 169.775244] niks3-server[1118]: 2026/09/23 05:14:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign7522worker1 # [ 169.784712] nix-grpc-daemon[1991]: time=2026-09-23T05:14:39.778Z level=INFO msg="Uploading 1 narinfos"7523lb # [ 169.782265] niks3-server[1118]: 2026/09/23 05:14:39 INFO Signed narinfos id=80 count=17524lb # [ 169.786936] niks3-server[1118]: 2026/09/23 05:14:39 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls7525lb # [ 169.800271] niks3-server[1118]: 2026/09/23 05:14:39 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete7526worker1 # [ 169.810743] nix-grpc-daemon[1991]: time=2026-09-23T05:14:39.804Z level=INFO msg="Upload complete. (180ms)"7527lb # [ 169.808468] niks3-server[1118]: 2026/09/23 05:14:39 INFO Completed upload id=807528lb # [ 169.811329] niks3-server[1118]: 2026/09/23 05:14:39 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo7529worker1 # [ 169.813864] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 paths=1 nar_bytes_in=6247530worker1 # [ 169.819689] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567531worker1 # [ 169.831071] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7532worker1 # [ 169.836099] nix-daemon[864]: reaped child process 2234, status = succeeded7533worker1 # [ 169.839907] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7534worker1 # [ 170.479510] nix-daemon[864]: reaped child process 2243, status = succeeded7535worker1 # [ 170.560624] nix-daemon[864]: accepted connection from pid 2259, user nix-grpc-daemon (trusted)7536worker1 # [ 170.575281] nix-daemon[864]: reaped child process 2262, status = succeeded7537lb # [ 170.582066] niks3-server[1118]: 2026/09/23 05:14:40 INFO Received uploads request method=POST path=/api/pending_closures7538worker1 # [ 170.592769] nix-grpc-daemon[1991]: time=2026-09-23T05:14:40.586Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7539worker1 # [ 170.595605] nix-grpc-daemon[1991]: time=2026-09-23T05:14:40.587Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"7540lb # [ 170.618462] niks3-server[1118]: 2026/09/23 05:14:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign7541worker1 # [ 170.627412] nix-grpc-daemon[1991]: time=2026-09-23T05:14:40.620Z level=INFO msg="Uploading 1 narinfos"7542lb # [ 170.624819] niks3-server[1118]: 2026/09/23 05:14:40 INFO Signed narinfos id=81 count=17543lb # [ 170.632657] niks3-server[1118]: 2026/09/23 05:14:40 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst7544lb # [ 170.642060] niks3-server[1118]: 2026/09/23 05:14:40 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls7545lb # [ 170.647099] niks3-server[1118]: 2026/09/23 05:14:40 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv7546lb # [ 170.654972] niks3-server[1118]: 2026/09/23 05:14:40 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete7547lb # [ 170.661131] niks3-server[1118]: 2026/09/23 05:14:40 INFO Completed upload id=817548worker1 # [ 170.663640] nix-grpc-daemon[1991]: time=2026-09-23T05:14:40.657Z level=INFO msg="Upload complete. (216ms)"7549worker1 # [ 170.669898] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=10 outputs=17550worker2 # [ 170.660670] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:40Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=17551worker1 # [ 170.679648] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567552worker2 # [ 170.666490] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:40Z level=debug event=want client=2 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=07553worker1 # [ 170.688787] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608567554worker1 # [ 170.698315] nix-daemon[864]: reaped child process 2241, status = succeeded7555worker1 # [ 170.699524] nix-grpc-daemon[1991]: time=2026-09-23T05:14:40.693Z level=INFO msg="All 2 paths already cached"7556worker1 # [ 170.704457] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608567557worker1 # [ 170.708276] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7558worker1 # [ 170.798954] nix-daemon[864]: accepted connection from pid 2266, user nix-grpc-daemon (trusted)7559worker1 # [ 170.813762] nix-daemon[864]: reaped child process 2269, status = succeeded7560lb # [ 170.818273] niks3-server[1118]: 2026/09/23 05:14:40 INFO Received uploads request method=POST path=/api/pending_closures7561worker1 # [ 170.830815] nix-grpc-daemon[1991]: time=2026-09-23T05:14:40.824Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7562worker1 # [ 170.832873] nix-grpc-daemon[1991]: time=2026-09-23T05:14:40.825Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"7563lb # [ 170.862342] niks3-server[1118]: 2026/09/23 05:14:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign7564worker1 # [ 170.868060] nix-grpc-daemon[1991]: time=2026-09-23T05:14:40.861Z level=INFO msg="Uploading 1 narinfos"7565lb # [ 170.865662] niks3-server[1118]: 2026/09/23 05:14:40 INFO Signed narinfos id=82 count=17566lb # [ 170.870497] niks3-server[1118]: 2026/09/23 05:14:40 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst7567lb # [ 170.876104] niks3-server[1118]: 2026/09/23 05:14:40 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls7568lb # [ 170.885535] niks3-server[1118]: 2026/09/23 05:14:40 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete7569lb # [ 170.888364] niks3-server[1118]: 2026/09/23 05:14:40 INFO Completed upload id=827570worker1 # [ 170.891050] nix-grpc-daemon[1991]: time=2026-09-23T05:14:40.884Z level=INFO msg="Upload complete. (176ms)"7571lb # [ 170.891099] niks3-server[1118]: 2026/09/23 05:14:40 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo7572worker1 # [ 170.894826] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 paths=1 nar_bytes_in=7527573worker1 # [ 170.900898] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567574worker1 # [ 170.910151] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7575worker1 # [ 170.917491] nix-daemon[864]: reaped child process 2264, status = succeeded7576worker1 # [ 170.920714] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7577worker1 # [ 171.534261] nix-daemon[864]: reaped child process 2273, status = succeeded7578worker1 # [ 171.611631] nix-daemon[864]: accepted connection from pid 2289, user nix-grpc-daemon (trusted)7579worker1 # [ 171.625418] nix-daemon[864]: reaped child process 2292, status = succeeded7580lb # [ 171.631478] niks3-server[1118]: 2026/09/23 05:14:41 INFO Received uploads request method=POST path=/api/pending_closures7581worker1 # [ 171.643678] nix-grpc-daemon[1991]: time=2026-09-23T05:14:41.637Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7582worker1 # [ 171.645497] nix-grpc-daemon[1991]: time=2026-09-23T05:14:41.638Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"7583lb # [ 171.674412] niks3-server[1118]: 2026/09/23 05:14:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign7584lb # [ 171.679113] niks3-server[1118]: 2026/09/23 05:14:41 INFO Signed narinfos id=83 count=17585worker1 # [ 171.685091] nix-grpc-daemon[1991]: time=2026-09-23T05:14:41.678Z level=INFO msg="Uploading 1 narinfos"7586lb # [ 171.690751] niks3-server[1118]: 2026/09/23 05:14:41 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls7587lb # [ 171.695193] niks3-server[1118]: 2026/09/23 05:14:41 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst7588lb # [ 171.705674] niks3-server[1118]: 2026/09/23 05:14:41 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv7589lb # [ 171.711896] niks3-server[1118]: 2026/09/23 05:14:41 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete7590lb # [ 171.715453] niks3-server[1118]: 2026/09/23 05:14:41 INFO Completed upload id=837591lb # [ 171.716949] niks3-server[1118]: 2026/09/23 05:14:41 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo7592worker1 # [ 171.718284] nix-grpc-daemon[1991]: time=2026-09-23T05:14:41.711Z level=INFO msg="Upload complete. (191ms)"7593worker2 # [ 171.711924] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:41Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=17594worker1 # [ 171.724923] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=11 outputs=17595worker1 # [ 171.740067] nix-daemon[864]: reaped child process 2271, status = succeeded7596worker1 # [ 171.769433] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567597worker1 # [ 171.782311] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608567598worker1 # [ 171.795808] nix-grpc-daemon[1991]: time=2026-09-23T05:14:41.789Z level=INFO msg="All 2 paths already cached"7599worker1 # [ 171.803834] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608567600worker1 # [ 171.814170] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7601worker1 # [ 171.911776] nix-daemon[864]: accepted connection from pid 2296, user nix-grpc-daemon (trusted)7602worker1 # [ 171.925405] nix-daemon[864]: reaped child process 2299, status = succeeded7603lb # [ 171.930915] niks3-server[1118]: 2026/09/23 05:14:41 INFO Received uploads request method=POST path=/api/pending_closures7604worker1 # [ 171.941321] nix-grpc-daemon[1991]: time=2026-09-23T05:14:41.934Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7605worker1 # [ 171.942878] nix-grpc-daemon[1991]: time=2026-09-23T05:14:41.936Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7606lb # [ 171.963142] niks3-server[1118]: 2026/09/23 05:14:41 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7607lb # [ 171.967805] niks3-server[1118]: 2026/09/23 05:14:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign7608worker1 # [ 171.978469] nix-grpc-daemon[1991]: time=2026-09-23T05:14:41.972Z level=INFO msg="Uploading 1 narinfos"7609lb # [ 171.975911] niks3-server[1118]: 2026/09/23 05:14:42 INFO Signed narinfos id=84 count=17610lb # [ 171.987939] niks3-server[1118]: 2026/09/23 05:14:42 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7611lb # [ 171.995875] niks3-server[1118]: 2026/09/23 05:14:42 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete7612lb # [ 172.000370] niks3-server[1118]: 2026/09/23 05:14:42 INFO Completed upload id=847613worker1 # [ 172.005220] nix-grpc-daemon[1991]: time=2026-09-23T05:14:41.998Z level=INFO msg="Upload complete. (179ms)"7614lb # [ 172.007567] niks3-server[1118]: 2026/09/23 05:14:42 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7615worker1 # [ 172.008986] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 paths=1 nar_bytes_in=7527616worker1 # [ 172.014353] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567617worker1 # [ 172.026768] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7618worker1 # [ 172.030349] nix-daemon[864]: reaped child process 2294, status = succeeded7619worker1 # [ 172.035384] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7620worker1 # [ 173.480705] nix-daemon[864]: reaped child process 2303, status = succeeded7621worker1 # [ 173.556147] nix-daemon[864]: accepted connection from pid 2319, user nix-grpc-daemon (trusted)7622worker1 # [ 173.570881] nix-daemon[864]: reaped child process 2322, status = succeeded7623lb # [ 173.575657] niks3-server[1118]: 2026/09/23 05:14:43 INFO Received uploads request method=POST path=/api/pending_closures7624worker1 # [ 173.587530] nix-grpc-daemon[1991]: time=2026-09-23T05:14:43.580Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7625worker1 # [ 173.589120] nix-grpc-daemon[1991]: time=2026-09-23T05:14:43.582Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7626lb # [ 173.615992] niks3-server[1118]: 2026/09/23 05:14:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign7627lb # [ 173.620340] niks3-server[1118]: 2026/09/23 05:14:43 INFO Signed narinfos id=85 count=17628worker1 # [ 173.627267] nix-grpc-daemon[1991]: time=2026-09-23T05:14:43.620Z level=INFO msg="Uploading 1 narinfos"7629lb # [ 173.629272] niks3-server[1118]: 2026/09/23 05:14:43 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7630lb # [ 173.635305] niks3-server[1118]: 2026/09/23 05:14:43 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7631lb # [ 173.642218] niks3-server[1118]: 2026/09/23 05:14:43 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete7632lb # [ 173.647588] niks3-server[1118]: 2026/09/23 05:14:43 INFO Completed upload id=857633lb # [ 173.649794] niks3-server[1118]: 2026/09/23 05:14:43 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7634worker1 # [ 173.651494] nix-grpc-daemon[1991]: time=2026-09-23T05:14:43.644Z level=INFO msg="Upload complete. (207ms)"7635lb # [ 173.652064] niks3-server[1118]: 2026/09/23 05:14:43 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7636worker1 # [ 173.656902] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=12 outputs=17637worker2 # [ 173.644731] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:43Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17638worker1 # [ 173.670491] nix-daemon[864]: reaped child process 2301, status = succeeded7639worker1 # [ 173.708690] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567640worker1 # [ 173.722252] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608567641worker1 # [ 173.736347] nix-grpc-daemon[1991]: time=2026-09-23T05:14:43.729Z level=INFO msg="All 2 paths already cached"7642worker1 # [ 173.744599] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608567643worker1 # [ 173.756173] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7644worker1 # [ 173.856332] nix-daemon[864]: accepted connection from pid 2326, user nix-grpc-daemon (trusted)7645worker1 # [ 173.870328] nix-daemon[864]: reaped child process 2329, status = succeeded7646lb # [ 173.875153] niks3-server[1118]: 2026/09/23 05:14:43 INFO Received uploads request method=POST path=/api/pending_closures7647worker1 # [ 173.886452] nix-grpc-daemon[1991]: time=2026-09-23T05:14:43.880Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7648worker1 # [ 173.888078] nix-grpc-daemon[1991]: time=2026-09-23T05:14:43.881Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7649lb # [ 173.910503] niks3-server[1118]: 2026/09/23 05:14:43 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7650lb # [ 173.913841] niks3-server[1118]: 2026/09/23 05:14:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/86/sign7651lb # [ 173.919250] niks3-server[1118]: 2026/09/23 05:14:43 INFO Signed narinfos id=86 count=17652worker1 # [ 173.924835] nix-grpc-daemon[1991]: time=2026-09-23T05:14:43.918Z level=INFO msg="Uploading 1 narinfos"7653lb # [ 173.927732] niks3-server[1118]: 2026/09/23 05:14:43 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7654lb # [ 173.936169] niks3-server[1118]: 2026/09/23 05:14:43 INFO Received complete upload request method=POST path=/api/pending_closures/86/complete7655lb # [ 173.938853] niks3-server[1118]: 2026/09/23 05:14:43 INFO Completed upload id=867656worker1 # [ 173.942503] nix-grpc-daemon[1991]: time=2026-09-23T05:14:43.936Z level=INFO msg="Upload complete. (173ms)"7657lb # [ 173.943746] niks3-server[1118]: 2026/09/23 05:14:43 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7658worker1 # [ 173.945745] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=0 paths=1 nar_bytes_in=7527659worker1 # [ 173.950705] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608567660worker1 # [ 173.962825] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7661worker1 # [ 173.966073] nix-daemon[864]: reaped child process 2324, status = succeeded7662worker1 # [ 173.970468] nix-daemon[864]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)7663worker1 # [ 174.807386] nix-daemon[864]: reaped child process 2333, status = succeeded7664worker1 # [ 174.891855] nix-daemon[864]: accepted connection from pid 2349, user nix-grpc-daemon (trusted)7665worker1 # [ 174.905454] nix-daemon[864]: reaped child process 2352, status = succeeded7666lb # [ 174.911693] niks3-server[1118]: 2026/09/23 05:14:44 INFO Received uploads request method=POST path=/api/pending_closures7667worker1 # [ 174.923555] nix-grpc-daemon[1991]: time=2026-09-23T05:14:44.916Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7668worker1 # [ 174.925121] nix-grpc-daemon[1991]: time=2026-09-23T05:14:44.918Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7669lb # [ 174.959631] niks3-server[1118]: 2026/09/23 05:14:44 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7670lb # [ 174.968896] niks3-server[1118]: 2026/09/23 05:14:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/87/sign7671lb # [ 174.974888] niks3-server[1118]: 2026/09/23 05:14:45 INFO Signed narinfos id=87 count=17672worker1 # [ 174.980069] nix-grpc-daemon[1991]: time=2026-09-23T05:14:44.973Z level=INFO msg="Uploading 1 narinfos"7673lb # [ 174.981226] niks3-server[1118]: 2026/09/23 05:14:45 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7674lb # [ 174.984844] niks3-server[1118]: 2026/09/23 05:14:45 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7675lb # [ 174.994598] niks3-server[1118]: 2026/09/23 05:14:45 INFO Received complete upload request method=POST path=/api/pending_closures/87/complete7676lb # [ 174.998851] niks3-server[1118]: 2026/09/23 05:14:45 INFO Completed upload id=877677worker1 # [ 175.001662] nix-grpc-daemon[1991]: time=2026-09-23T05:14:44.995Z level=INFO msg="Upload complete. (224ms)"7678lb # [ 175.001609] niks3-server[1118]: 2026/09/23 05:14:45 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7679worker1 # [ 175.008220] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60856 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=13 outputs=17680worker1 # [ 175.023269] nix-daemon[864]: reaped child process 2331, status = succeeded7681client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 13.08 seconds)7682worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7683worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7684client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]'7685client # [ 175.135630] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7686client # Running as unit: gpu.service; invocation ID: fbfa712f3e78430aa3a9bb64cfc8aea87687client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.06 seconds)7688lb: must succeed: curl -sf localhost:9901/clusters7689lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)7690lb: must succeed: curl -sf localhost:9901/clusters7691lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7692worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7693client # [ 175.272552] nix[1578]: don't know how to build these paths:7694client # [ 175.273426] nix[1578]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7695worker2 # [ 175.261336] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:374507696worker1 # [ 175.277083] nix-grpc-daemon[1926]: ts=2026-09-23T05:14:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608567697client # [ 175.293536] nix[1578]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,kvm,nixos-test,vip}, waiting up to 30s for a worker7698worker2 # [ 175.281989] nix-grpc-daemon[1436]: ts=2026-09-23T05:14:45Z level=debug event=want client=3 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07699worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7700client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7701lb # [ 184.883555] envoy[903]: conn peer=[2001:db8:1::1]:58612 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27133 tx=12291 ms=129047702client # [ 205.294866] nix[1578]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,kvm,nixos-test,vip}7703client # [ 205.306267] nix[1578]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7704client # [ 205.331706] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7705client # [ 205.335516] systemd[1]: gpu.service: Failed with result 'exit-code'.7706client # [ 205.340144] systemd[1]: gpu.service: Consumed 144ms CPU time over 30.199s wall clock time, 37.6M memory peak, 5.6K incoming IP traffic, 5.3K outgoing IP traffic.7707client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 30.43 seconds)7708client: must succeed: journalctl -u gpu -o cat7709client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds)7710lb: must succeed: curl -sf localhost:9901/clusters7711lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7712lb: must succeed: curl -sf localhost:9901/clusters7713lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7714worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7715worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7716(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 43.90 seconds)7717subtest: low disk drains a worker and builds go to the other7718worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7719worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.02 seconds)7720worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7721worker1 # [ 207.395494] nix-grpc-daemon[1926]: ts=2026-09-23T05:15:17Z level=info event=unhealthy reason=min_free7722worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.08 seconds)7723client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&27724worker2 # [ 208.100698] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:18Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:374507725client # don't know how to build these paths:7726client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7727worker2 # [ 208.109152] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374507728worker2 # [ 208.130900] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:18Z level=debug event=want client=4 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07729worker2 # [ 208.136634] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:18Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17730worker2 # [ 208.175321] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374507731worker2 # [ 208.188234] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374507732client # copying 1 paths to 'lb:50051'...7733client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7734worker2 # [ 208.201542] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374507735worker2 # [ 208.213385] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7736worker2 # [ 208.317102] nix-daemon[851]: accepted connection from pid 1819, user nix-grpc-daemon (trusted)7737worker2 # [ 208.330757] nix-daemon[851]: reaped child process 1822, status = succeeded7738lb # [ 208.349556] niks3-server[1118]: 2026/09/23 05:15:18 INFO Received uploads request method=POST path=/api/pending_closures7739worker2 # [ 208.346581] nix-grpc-daemon[1525]: time=2026-09-23T05:15:18.511Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7740worker2 # [ 208.348749] nix-grpc-daemon[1525]: time=2026-09-23T05:15:18.513Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7741lb # [ 208.385137] niks3-server[1118]: 2026/09/23 05:15:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/88/sign7742worker2 # [ 208.380134] nix-grpc-daemon[1525]: time=2026-09-23T05:15:18.545Z level=INFO msg="Uploading 1 narinfos"7743lb # [ 208.391618] niks3-server[1118]: 2026/09/23 05:15:18 INFO Signed narinfos id=88 count=17744lb # [ 208.403548] niks3-server[1118]: 2026/09/23 05:15:18 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7745lb # [ 208.409056] niks3-server[1118]: 2026/09/23 05:15:18 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7746lb # [ 208.416662] niks3-server[1118]: 2026/09/23 05:15:18 INFO Received complete upload request method=POST path=/api/pending_closures/88/complete7747lb # [ 208.420716] niks3-server[1118]: 2026/09/23 05:15:18 INFO Completed upload id=887748worker2 # [ 208.409450] nix-grpc-daemon[1525]: time=2026-09-23T05:15:18.574Z level=INFO msg="Upload complete. (184ms)"7749worker2 # [ 208.413478] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=0 paths=1 nar_bytes_in=5127750lb # [ 208.429508] niks3-server[1118]: 2026/09/23 05:15:18 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7751worker2 # [ 208.418775] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374507752worker2 # [ 208.431041] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7753worker2 # [ 208.434434] nix-daemon[851]: reaped child process 1816, status = succeeded7754worker2 # [ 208.439771] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7755client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7756client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7757client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7758client # farm-a> LOG-farm-a-drain7759worker2 # [ 209.667355] nix-daemon[851]: reaped child process 1826, status = succeeded7760worker2 # [ 209.740475] nix-daemon[851]: accepted connection from pid 1842, user nix-grpc-daemon (trusted)7761worker2 # [ 209.753476] nix-daemon[851]: reaped child process 1845, status = succeeded7762lb # [ 209.773501] niks3-server[1118]: 2026/09/23 05:15:19 INFO Received uploads request method=POST path=/api/pending_closures7763worker2 # [ 209.772374] nix-grpc-daemon[1525]: time=2026-09-23T05:15:19.937Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7764worker2 # [ 209.774264] nix-grpc-daemon[1525]: time=2026-09-23T05:15:19.939Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7765lb # [ 209.821908] niks3-server[1118]: 2026/09/23 05:15:19 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7766lb # [ 209.828364] niks3-server[1118]: 2026/09/23 05:15:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/89/sign7767lb # [ 209.836293] niks3-server[1118]: 2026/09/23 05:15:19 INFO Signed narinfos id=89 count=17768worker2 # [ 209.824903] nix-grpc-daemon[1525]: time=2026-09-23T05:15:19.990Z level=INFO msg="Uploading 1 narinfos"7769lb # [ 209.838577] niks3-server[1118]: 2026/09/23 05:15:19 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7770lb # [ 209.841571] niks3-server[1118]: 2026/09/23 05:15:19 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7771lb # [ 209.846862] niks3-server[1118]: 2026/09/23 05:15:19 INFO Received complete upload request method=POST path=/api/pending_closures/89/complete7772lb # [ 209.850387] niks3-server[1118]: 2026/09/23 05:15:19 INFO Completed upload id=897773worker2 # [ 209.838998] nix-grpc-daemon[1525]: time=2026-09-23T05:15:20.004Z level=INFO msg="Upload complete. (192ms)"7774lb # [ 209.854415] niks3-server[1118]: 2026/09/23 05:15:19 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7775worker2 # [ 209.847202] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=14 outputs=17776worker2 # [ 209.856981] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:20Z level=debug event=want client=4 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07777worker2 # [ 209.862226] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:20Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17778worker2 # [ 209.870350] nix-daemon[851]: reaped child process 1824, status = succeeded7779worker2 # [ 209.901094] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374507780worker2 # [ 209.913521] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374507781worker2 # [ 209.925874] nix-grpc-daemon[1525]: time=2026-09-23T05:15:20.091Z level=INFO msg="All 1 paths already cached"7782client # copying 1 paths to 'lb:50051'...7783client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7784worker2 # [ 209.934805] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374507785worker2 # [ 209.945190] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7786worker2 # [ 210.042947] nix-daemon[851]: accepted connection from pid 1851, user nix-grpc-daemon (trusted)7787worker2 # [ 210.056091] nix-daemon[851]: reaped child process 1854, status = succeeded7788lb # [ 210.075777] niks3-server[1118]: 2026/09/23 05:15:20 INFO Received uploads request method=POST path=/api/pending_closures7789worker2 # [ 210.073444] nix-grpc-daemon[1525]: time=2026-09-23T05:15:20.238Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7790worker2 # [ 210.075092] nix-grpc-daemon[1525]: time=2026-09-23T05:15:20.240Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7791lb # [ 210.110643] niks3-server[1118]: 2026/09/23 05:15:20 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7792lb # [ 210.114555] niks3-server[1118]: 2026/09/23 05:15:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/90/sign7793lb # [ 210.119044] niks3-server[1118]: 2026/09/23 05:15:20 INFO Signed narinfos id=90 count=17794worker2 # [ 210.109697] nix-grpc-daemon[1525]: time=2026-09-23T05:15:20.274Z level=INFO msg="Uploading 1 narinfos"7795lb # [ 210.127956] niks3-server[1118]: 2026/09/23 05:15:20 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7796lb # [ 210.137065] niks3-server[1118]: 2026/09/23 05:15:20 INFO Received complete upload request method=POST path=/api/pending_closures/90/complete7797lb # [ 210.140666] niks3-server[1118]: 2026/09/23 05:15:20 INFO Completed upload id=907798worker2 # [ 210.130381] nix-grpc-daemon[1525]: time=2026-09-23T05:15:20.295Z level=INFO msg="Upload complete. (174ms)"7799lb # [ 210.142636] niks3-server[1118]: 2026/09/23 05:15:20 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7800worker2 # [ 210.133374] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=0 paths=1 nar_bytes_in=6247801worker2 # [ 210.138765] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374507802worker2 # [ 210.151072] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7803worker2 # [ 210.153730] nix-daemon[851]: reaped child process 1849, status = succeeded7804worker2 # [ 210.157790] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7805client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7806client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7807client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7808client # farm-b> LOG-farm-b-drain7809worker2 # [ 211.196605] nix-daemon[851]: reaped child process 1858, status = succeeded7810worker2 # [ 211.276988] nix-daemon[851]: accepted connection from pid 1874, user nix-grpc-daemon (trusted)7811worker2 # [ 211.288978] nix-daemon[851]: reaped child process 1877, status = succeeded7812lb # [ 211.309244] niks3-server[1118]: 2026/09/23 05:15:21 INFO Received uploads request method=POST path=/api/pending_closures7813worker2 # [ 211.307556] nix-grpc-daemon[1525]: time=2026-09-23T05:15:21.472Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7814worker2 # [ 211.310056] nix-grpc-daemon[1525]: time=2026-09-23T05:15:21.474Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7815lb # [ 211.358948] niks3-server[1118]: 2026/09/23 05:15:21 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7816lb # [ 211.363141] niks3-server[1118]: 2026/09/23 05:15:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/91/sign7817lb # [ 211.367867] niks3-server[1118]: 2026/09/23 05:15:21 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7818lb # [ 211.375257] niks3-server[1118]: 2026/09/23 05:15:21 INFO Signed narinfos id=91 count=17819worker2 # [ 211.367724] nix-grpc-daemon[1525]: time=2026-09-23T05:15:21.532Z level=INFO msg="Uploading 1 narinfos"7820lb # [ 211.382112] niks3-server[1118]: 2026/09/23 05:15:21 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7821lb # [ 211.391905] niks3-server[1118]: 2026/09/23 05:15:21 INFO Received complete upload request method=POST path=/api/pending_closures/91/complete7822lb # [ 211.396864] niks3-server[1118]: 2026/09/23 05:15:21 INFO Completed upload id=917823lb # [ 211.398760] niks3-server[1118]: 2026/09/23 05:15:21 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7824worker2 # [ 211.386586] nix-grpc-daemon[1525]: time=2026-09-23T05:15:21.551Z level=INFO msg="Upload complete. (221ms)"7825worker2 # [ 211.396219] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=15 outputs=17826worker2 # [ 211.405750] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:21Z level=debug event=want client=4 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07827worker2 # [ 211.410406] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:21Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17828worker2 # [ 211.420680] nix-daemon[851]: reaped child process 1856, status = succeeded7829worker2 # [ 211.450185] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374507830worker2 # [ 211.463553] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:374507831worker2 # [ 211.477971] nix-grpc-daemon[1525]: time=2026-09-23T05:15:21.643Z level=INFO msg="All 2 paths already cached"7832client # copying 1 paths to 'lb:50051'...7833client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7834worker2 # [ 211.487937] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:374507835worker2 # [ 211.496795] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7836worker2 # [ 211.598498] nix-daemon[851]: accepted connection from pid 1883, user nix-grpc-daemon (trusted)7837worker2 # [ 211.613795] nix-daemon[851]: reaped child process 1886, status = succeeded7838lb # [ 211.632228] niks3-server[1118]: 2026/09/23 05:15:21 INFO Received uploads request method=POST path=/api/pending_closures7839worker2 # [ 211.628763] nix-grpc-daemon[1525]: time=2026-09-23T05:15:21.794Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7840worker2 # [ 211.630347] nix-grpc-daemon[1525]: time=2026-09-23T05:15:21.795Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7841lb # [ 211.665378] niks3-server[1118]: 2026/09/23 05:15:21 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7842lb # [ 211.671260] niks3-server[1118]: 2026/09/23 05:15:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/92/sign7843lb # [ 211.678633] niks3-server[1118]: 2026/09/23 05:15:21 INFO Signed narinfos id=92 count=17844worker2 # [ 211.670594] nix-grpc-daemon[1525]: time=2026-09-23T05:15:21.835Z level=INFO msg="Uploading 1 narinfos"7845lb # [ 211.686899] niks3-server[1118]: 2026/09/23 05:15:21 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7846lb # [ 211.698293] niks3-server[1118]: 2026/09/23 05:15:21 INFO Received complete upload request method=POST path=/api/pending_closures/92/complete7847lb # [ 211.706605] niks3-server[1118]: 2026/09/23 05:15:21 INFO Completed upload id=927848worker2 # [ 211.695067] nix-grpc-daemon[1525]: time=2026-09-23T05:15:21.860Z level=INFO msg="Upload complete. (185ms)"7849lb # [ 211.710805] niks3-server[1118]: 2026/09/23 05:15:21 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7850worker2 # [ 211.699234] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=0 paths=1 nar_bytes_in=7527851worker2 # [ 211.706541] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:374507852worker2 # [ 211.717313] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7853worker2 # [ 211.725182] nix-daemon[851]: reaped child process 1881, status = succeeded7854worker2 # [ 211.730298] nix-daemon[851]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)7855client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7856client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7857client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7858client # farm-top> LOG-farm-top-drain7859worker2 # [ 212.769510] nix-daemon[851]: reaped child process 1890, status = succeeded7860worker2 # [ 212.850934] nix-daemon[851]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)7861worker2 # [ 212.863582] nix-daemon[851]: reaped child process 1909, status = succeeded7862lb # [ 212.884073] niks3-server[1118]: 2026/09/23 05:15:22 INFO Received uploads request method=POST path=/api/pending_closures7863worker2 # [ 212.881986] nix-grpc-daemon[1525]: time=2026-09-23T05:15:23.047Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7864worker2 # [ 212.884395] nix-grpc-daemon[1525]: time=2026-09-23T05:15:23.048Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7865lb # [ 212.925116] niks3-server[1118]: 2026/09/23 05:15:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/93/sign7866worker2 # [ 212.919195] nix-grpc-daemon[1525]: time=2026-09-23T05:15:23.084Z level=INFO msg="Uploading 1 narinfos"7867lb # [ 212.931473] niks3-server[1118]: 2026/09/23 05:15:22 INFO Signed narinfos id=93 count=17868lb # [ 212.939724] niks3-server[1118]: 2026/09/23 05:15:22 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7869lb # [ 212.947407] niks3-server[1118]: 2026/09/23 05:15:22 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7870lb # [ 212.951691] niks3-server[1118]: 2026/09/23 05:15:22 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7871lb # [ 212.962147] niks3-server[1118]: 2026/09/23 05:15:22 INFO Received complete upload request method=POST path=/api/pending_closures/93/complete7872lb # [ 212.964874] niks3-server[1118]: 2026/09/23 05:15:22 INFO Completed upload id=937873worker2 # [ 212.953568] nix-grpc-daemon[1525]: time=2026-09-23T05:15:23.118Z level=INFO msg="Upload complete. (215ms)"7874lb # [ 212.969418] niks3-server[1118]: 2026/09/23 05:15:22 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7875worker2 # [ 212.959544] nix-grpc-daemon[1436]: ts=2026-09-23T05:15:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37450 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=16 outputs=17876worker2 # [ 212.976507] nix-daemon[851]: reaped child process 1888, status = succeeded7877client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&2, in 5.04 seconds)7878client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath7879client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath, in 0.11 seconds)7880worker1: must succeed: rm /nix/.rw-store/fill7881worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.12 seconds)7882worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7883worker1 # [ 213.397524] nix-grpc-daemon[1926]: ts=2026-09-23T05:15:23Z level=info event=healthy reason=min_free7884worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.08 seconds)7885(finished: subtest: low disk drains a worker and builds go to the other, in 8.50 seconds)7886(finished: run the VM test script, in 215.28 seconds)7887test script finished in 215.45s7888cleanup7889kill QemuMachine (pid 50)7890client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7891client # [2026-09-23T05:15:24Z INFO virtiofsd] Client disconnected, shutting down7892client # [2026-09-23T05:15:24Z INFO virtiofsd] Client disconnected, shutting down7893client # [2026-09-23T05:15:24Z INFO virtiofsd] Client disconnected, shutting down7894kill QemuMachine (pid 49)7895lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7896lb # [2026-09-23T05:15:25Z INFO virtiofsd] Client disconnected, shutting down7897lb # [2026-09-23T05:15:25Z INFO virtiofsd] Client disconnected, shutting down7898lb # [2026-09-23T05:15:25Z INFO virtiofsd] Client disconnected, shutting down7899kill QemuMachine (pid 51)7900worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7901worker1 # [2026-09-23T05:15:25Z INFO virtiofsd] Client disconnected, shutting down7902worker1 # [2026-09-23T05:15:25Z INFO virtiofsd] Client disconnected, shutting down7903worker1 # [2026-09-23T05:15:25Z INFO virtiofsd] Client disconnected, shutting down7904kill QemuMachine (pid 52)7905worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7906worker2 # [2026-09-23T05:15:25Z INFO virtiofsd] Client disconnected, shutting down7907worker2 # [2026-09-23T05:15:25Z INFO virtiofsd] Client disconnected, shutting down7908worker2 # [2026-09-23T05:15:25Z INFO virtiofsd] Client disconnected, shutting down7909(finished: cleanup, in 1.02 seconds)7910additionally exposed symbols:7911 client, lb, worker1, worker2,7912 vlan1,7913 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh