vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #498
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12client # Disk image does not exist, creating the virtualisation disk image...13client # Formatting '/build/vm-state-client/tmp.mQYsSxmsbr', fmt=raw size=107374182414client: QEMU running (pid 49)15lb # Disk image does not exist, creating the virtualisation disk image...16lb: QEMU running (pid 50)17lb # Formatting '/build/vm-state-lb/tmp.4ro5bdNmLV', fmt=raw size=107374182418client # mke2fs 1.47.4 (6-Mar-2025)19lb # mke2fs 1.47.4 (6-Mar-2025)20client # Discarding device blocks: 0/262144 done21lb # Discarding device blocks: 0/262144 done22client # Creating filesystem with 262144 4k blocks and 65536 inodes23lb # Creating filesystem with 262144 4k blocks and 65536 inodes24client # Filesystem UUID: 95830c48-09f8-4980-a821-1a3c2658b02c25lb # Filesystem UUID: 86e65e3a-d71a-476b-99c1-afe1eaa27d7926client # Superblock backups stored on blocks:27lb # Superblock backups stored on blocks:28client # 32768, 98304, 163840, 22937629lb # 32768, 98304, 163840, 22937630client # 31lb # 32client # Allocating group tables: 0/8 done33lb # Allocating group tables: 0/8 done34client # Writing inode tables: 0/8 done35lb # Writing inode tables: 0/8 done36client # Creating journal (8192 blocks): done37lb # Creating journal (8192 blocks): done38client # Writing superblocks and filesystem accounting information: 0/8 done39lb # Writing superblocks and filesystem accounting information: 0/8 done40client # 41lb # 42client # Virtualisation disk image created.43lb # Virtualisation disk image created.44client # Starting virtiofs daemons...45lb # Starting virtiofs daemons...46client # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)47worker2: QEMU running (pid 52)48client # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether49worker1 # Disk image does not exist, creating the virtualisation disk image...50worker1: QEMU running (pid 51)51client # [2026-09-22T13:45:14Z INFO virtiofsd] Waiting for vhost-user socket connection...52worker1 # Formatting '/build/vm-state-worker1/tmp.kmDfEb3xJd', fmt=raw size=107374182453lb # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)54worker2 # Disk image does not exist, creating the virtualisation disk image...55client # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)56client # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether57lb # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether58lb # [2026-09-22T13:45:14Z INFO virtiofsd] Waiting for vhost-user socket connection...59lb # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)60lb # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether61lb # [2026-09-22T13:45:14Z INFO virtiofsd] Waiting for vhost-user socket connection...62lb # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)63(finished: start all VMs, in 0.23 seconds)64worker1 # mke2fs 1.47.4 (6-Mar-2025)65worker2 # Formatting '/build/vm-state-worker2/tmp.irL26K5AEK', fmt=raw size=107374182466client: waiting for unit network-addresses-eth1.service67worker1 # Discarding device blocks: 0/262144 done68worker2 # mke2fs 1.47.4 (6-Mar-2025)69client: waiting for the VM to finish booting70worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes71worker2 # Discarding device blocks: 0/262144 done72client # [2026-09-22T13:45:14Z INFO virtiofsd] Waiting for vhost-user socket connection...73lb # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether74worker1 # Filesystem UUID: 7129ed64-7510-46d7-bff0-0b537e13403b75worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes76client # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)77lb # [2026-09-22T13:45:14Z INFO virtiofsd] Waiting for vhost-user socket connection...78worker1 # Superblock backups stored on blocks:79worker2 # Filesystem UUID: 2679bd0b-f316-4cc7-b526-db7c0c2f47d480client # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether81lb # [2026-09-22T13:45:14Z INFO virtiofsd] Client connected, servicing requests82worker1 # 32768, 98304, 163840, 22937683worker2 # Superblock backups stored on blocks:84client # [2026-09-22T13:45:14Z INFO virtiofsd] Waiting for vhost-user socket connection...85client # [2026-09-22T13:45:14Z INFO virtiofsd] Client connected, servicing requests86lb # [2026-09-22T13:45:14Z INFO virtiofsd] Client connected, servicing requests87worker2 # 32768, 98304, 163840, 22937688lb # [2026-09-22T13:45:14Z INFO virtiofsd] Client connected, servicing requests89worker1 # 90client # [2026-09-22T13:45:14Z INFO virtiofsd] Client connected, servicing requests91worker1 # Allocating group tables: 0/8 done92client # [2026-09-22T13:45:14Z INFO virtiofsd] Client connected, servicing requests93worker1 # Writing inode tables: 0/8 done94worker2 # 95worker1 # Creating journal (8192 blocks): done96worker2 # Allocating group tables: 0/8 done97worker1 # Writing superblocks and filesystem accounting information: 0/8 done98worker2 # Writing inode tables: 0/8 done99worker1 # 100worker2 # Creating journal (8192 blocks): done101worker1 # Virtualisation disk image created.102worker2 # Writing superblocks and filesystem accounting information: 0/8 done103worker1 # Starting virtiofs daemons...104worker2 # 105worker1 # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)106worker2 # Virtualisation disk image created.107worker1 # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether108worker2 # Starting virtiofs daemons...109worker1 # [2026-09-22T13:45:14Z INFO virtiofsd] Waiting for vhost-user socket connection...110worker2 # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)111worker1 # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)112worker2 # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether113worker1 # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether114worker2 # [2026-09-22T13:45:14Z INFO virtiofsd] Waiting for vhost-user socket connection...115worker1 # [2026-09-22T13:45:14Z INFO virtiofsd] Waiting for vhost-user socket connection...116worker2 # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)117worker1 # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)118worker2 # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether119worker1 # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether120worker2 # [2026-09-22T13:45:14Z INFO virtiofsd] Waiting for vhost-user socket connection...121worker1 # [2026-09-22T13:45:14Z INFO virtiofsd] Waiting for vhost-user socket connection...122worker2 # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)123worker1 # [2026-09-22T13:45:14Z INFO virtiofsd] Client connected, servicing requests124worker2 # [2026-09-22T13:45:14Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether125worker1 # [2026-09-22T13:45:14Z INFO virtiofsd] Client connected, servicing requests126worker2 # [2026-09-22T13:45:14Z INFO virtiofsd] Waiting for vhost-user socket connection...127worker1 # [2026-09-22T13:45:14Z INFO virtiofsd] Client connected, servicing requests128worker2 # [2026-09-22T13:45:14Z INFO virtiofsd] Client connected, servicing requests129worker2 # [2026-09-22T13:45:14Z INFO virtiofsd] Client connected, servicing requests130worker2 # [2026-09-22T13:45:14Z INFO virtiofsd] Client connected, servicing requests131client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)132worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)133worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)134lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)135client # 136client # 137client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00138client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...139client # 140client # 141client # 142client # 143client # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00144worker2 # 145worker2 # 146worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00147client # Press Ctrl-B to configure iPXE (PCI 00:05.0)...148client # 149client # 150worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...151worker2 # 152worker2 # 153worker2 # 154worker2 # 155worker2 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00156worker2 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...157worker2 # 158worker2 # 159lb # 160lb # 161worker1 # 162worker1 # 163lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00164worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00165lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...166lb # 167lb # 168lb # 169lb # 170worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...171worker1 # 172worker1 # 173worker1 # 174worker1 # 175lb # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00176worker1 # iPXE (http://ipxe.org) 00:05.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00177lb # Press Ctrl-B to configure iPXE (PCI 00:05.0)...178lb # 179lb # 180worker1 # Press Ctrl-B to configure iPXE (PCI 00:05.0)...181worker1 # 182worker1 # 183client # Booting from ROM...184worker2 # Booting from ROM...185worker1 # Booting from ROM...186lb # Booting from ROM...187client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026188client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/sj30989cgkidyy8h42lc8n59b9vy41qh-nixos-system-client-test/init regInfo=/nix/store/j0bs04qrv7kdcmc859n70b50kc7sz3gd-closure-info/registration console=ttyS0,115200n8 console=tty0189client # [ 0.000000] BIOS-provided physical RAM map:190client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable191client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved192client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved193client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable194client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved195client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved196client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved197worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026198client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved199client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved200client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved201client # [ 0.000000] NX (Execute Disable) protection: active202client # [ 0.000000] APIC: Static calls initialized203client # [ 0.000000] SMBIOS 2.8 present.204worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/lvqq63fax2ya6fsm3d7s7hfg0rmfbnl4-nixos-system-worker2-test/init regInfo=/nix/store/6pd0cf3aa37mjz7663lg8n7pgkfbwknl-closure-info/registration console=ttyS0,115200n8 console=tty0205worker2 # [ 0.000000] BIOS-provided physical RAM map:206client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014207client # [ 0.000000] DMI: Memory slots populated: 1/1208worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable209client # [ 0.000000] Hypervisor detected: KVM210worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved211client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000212client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00213worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved214client # [ 0.000000] kvm-clock: using sched offset of 474330809 cycles215worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable216worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved217client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns218worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved219client # [ 0.000005] tsc: Detected 2400.012 MHz processor220client # [ 0.000804] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000221worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved222worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved223client # [ 0.000837] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs224worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved225client # [ 0.000839] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT226client # [ 0.002719] found SMP MP-table at [mem 0x000f5450-0x000f545f]227worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved228client # [ 0.002731] Using GB pages for direct mapping229worker2 # [ 0.000000] NX (Execute Disable) protection: active230client # [ 0.002822] RAMDISK: [mem 0x5e370000-0x5ffcffff]231worker2 # [ 0.000000] APIC: Static calls initialized232worker2 # [ 0.000000] SMBIOS 2.8 present.233client # [ 0.002838] ACPI: Early table checksum verification disabled234client # [ 0.002840] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )235worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014236worker2 # [ 0.000000] DMI: Memory slots populated: 1/1237worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026238client # [ 0.002844] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)239worker2 # [ 0.000000] Hypervisor detected: KVM240worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000241client # [ 0.002849] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)242worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00243client # [ 0.002855] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)244worker2 # [ 0.000000] kvm-clock: using sched offset of 470558837 cycles245client # [ 0.002857] ACPI: FACS 0x000000005FFE0000 000040246worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/dn8gqamk2zyr2nvpgiwfbpgwzjyrjlpw-nixos-system-worker1-test/init regInfo=/nix/store/ndzr4x0h49369zkyn79cnanic26lxrvk-closure-info/registration console=ttyS0,115200n8 console=tty0247worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns248client # [ 0.002859] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)249worker1 # [ 0.000000] BIOS-provided physical RAM map:250worker2 # [ 0.000004] tsc: Detected 2400.012 MHz processor251client # [ 0.002860] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)252worker2 # [ 0.000810] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000253worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable254client # [ 0.002862] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)255worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved256worker2 # [ 0.000837] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs257lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 14 11:36:19 UTC 2026258worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved259client # [ 0.002863] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)260worker2 # [ 0.000839] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT261worker2 # [ 0.002728] found SMP MP-table at [mem 0x000f5450-0x000f545f]262worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable263client # [ 0.002865] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]264worker2 # [ 0.002738] Using GB pages for direct mapping265worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved266worker2 # [ 0.002856] RAMDISK: [mem 0x5e36f000-0x5ffcffff]267client # [ 0.002866] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]268worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved269worker2 # [ 0.002866] ACPI: Early table checksum verification disabled270client # [ 0.002866] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]271worker2 # [ 0.002869] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )272worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved273lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/qfr7fghmwm537hb3f4f5is3yb8phm6jx-nixos-system-lb-test/init regInfo=/nix/store/s1480xaqc00gh7hz31b4z2fwjpnxi3sa-closure-info/registration console=ttyS0,115200n8 console=tty0274client # [ 0.002867] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]275lb # [ 0.000000] BIOS-provided physical RAM map:276worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved277worker2 # [ 0.002873] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)278client # [ 0.002867] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]279lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable280worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved281client # [ 0.002868] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]282worker2 # [ 0.002877] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)283lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved284worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved285client # [ 0.002868] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]286worker2 # [ 0.002885] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)287lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved288worker1 # [ 0.000000] NX (Execute Disable) protection: active289client # [ 0.003093] No NUMA configuration found290worker2 # [ 0.002887] ACPI: FACS 0x000000005FFE0000 000040291worker1 # [ 0.000000] APIC: Static calls initialized292lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable293worker1 # [ 0.000000] SMBIOS 2.8 present.294client # [ 0.003094] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]295worker2 # [ 0.002888] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)296lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved297client # [ 0.003096] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]298client # [ 0.005586] Zone ranges:299worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014300lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved301worker2 # [ 0.002890] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)302worker1 # [ 0.000000] DMI: Memory slots populated: 1/1303client # [ 0.005587] DMA [mem 0x0000000000001000-0x0000000000ffffff]304worker1 # [ 0.000000] Hypervisor detected: KVM305lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved306worker2 # [ 0.002891] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)307client # [ 0.005588] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]308worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000309client # [ 0.005589] Normal empty310lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved311client # [ 0.005590] Device empty312worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00313worker2 # [ 0.002893] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)314client # [ 0.005590] Movable zone start for each node315lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved316worker1 # [ 0.000000] kvm-clock: using sched offset of 499658053 cycles317client # [ 0.005591] Early memory node ranges318worker2 # [ 0.002894] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]319lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved320client # [ 0.005592] node 0: [mem 0x0000000000001000-0x000000000009efff]321worker2 # [ 0.002895] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]322worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns323lb # [ 0.000000] NX (Execute Disable) protection: active324client # [ 0.005593] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]325worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor326lb # [ 0.000000] APIC: Static calls initialized327worker2 # [ 0.002896] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]328lb # [ 0.000000] SMBIOS 2.8 present.329client # [ 0.005594] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]330worker1 # [ 0.000813] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000331worker2 # [ 0.002896] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]332client # [ 0.005614] On node 0, zone DMA: 1 pages in unavailable ranges333lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014334worker1 # [ 0.000840] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs335worker2 # [ 0.002897] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]336client # [ 0.005902] On node 0, zone DMA: 97 pages in unavailable ranges337lb # [ 0.000000] DMI: Memory slots populated: 1/1338lb # [ 0.000000] Hypervisor detected: KVM339worker1 # [ 0.000842] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT340worker2 # [ 0.002897] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]341client # [ 0.036099] On node 0, zone DMA32: 40 pages in unavailable ranges342worker1 # [ 0.002727] found SMP MP-table at [mem 0x000f5450-0x000f545f]343lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000344client # [ 0.036566] ACPI: PM-Timer IO Port: 0x608345worker2 # [ 0.002898] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]346worker1 # [ 0.002738] Using GB pages for direct mapping347lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00348worker2 # [ 0.003136] No NUMA configuration found349client # [ 0.036581] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])350worker1 # [ 0.002898] RAMDISK: [mem 0x5e370000-0x5ffcffff]351lb # [ 0.000000] kvm-clock: using sched offset of 498145849 cycles352worker2 # [ 0.003137] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]353client # [ 0.036607] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23354worker1 # [ 0.002906] ACPI: Early table checksum verification disabled355worker2 # [ 0.003140] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]356client # [ 0.036610] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)357worker1 # [ 0.002909] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )358lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns359worker2 # [ 0.005542] Zone ranges:360lb # [ 0.000005] tsc: Detected 2400.012 MHz processor361client # [ 0.036612] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)362worker2 # [ 0.005543] DMA [mem 0x0000000000001000-0x0000000000ffffff]363worker1 # [ 0.002913] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)364lb # [ 0.000816] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000365client # [ 0.036613] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)366worker2 # [ 0.005544] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]367worker2 # [ 0.005545] Normal empty368worker1 # [ 0.002917] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)369worker2 # [ 0.005546] Device empty370lb # [ 0.000843] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs371client # [ 0.036614] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)372worker2 # [ 0.005546] Movable zone start for each node373worker1 # [ 0.002924] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)374lb # [ 0.000846] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT375client # [ 0.036615] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)376worker2 # [ 0.005547] Early memory node ranges377worker1 # [ 0.002926] ACPI: FACS 0x000000005FFE0000 000040378lb # [ 0.002730] found SMP MP-table at [mem 0x000f5450-0x000f545f]379client # [ 0.036618] ACPI: Using ACPI (MADT) for SMP configuration information380worker2 # [ 0.005547] node 0: [mem 0x0000000000001000-0x000000000009efff]381lb # [ 0.002741] Using GB pages for direct mapping382client # [ 0.036619] ACPI: HPET id: 0x8086a201 base: 0xfed00000383worker1 # [ 0.002927] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)384worker2 # [ 0.005548] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]385lb # [ 0.002878] RAMDISK: [mem 0x5e36e000-0x5ffcffff]386client # [ 0.036623] TSC deadline timer available387lb # [ 0.002885] ACPI: Early table checksum verification disabled388client # [ 0.036627] CPU topo: Max. logical packages: 1389worker1 # [ 0.002929] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)390worker2 # [ 0.005549] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]391client # [ 0.036628] CPU topo: Max. logical dies: 1392lb # [ 0.002889] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )393worker2 # [ 0.005573] On node 0, zone DMA: 1 pages in unavailable ranges394client # [ 0.036628] CPU topo: Max. dies per package: 1395worker1 # [ 0.002930] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)396worker2 # [ 0.005849] On node 0, zone DMA: 97 pages in unavailable ranges397lb # [ 0.002892] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)398client # [ 0.036632] CPU topo: Max. threads per core: 1399worker1 # [ 0.002932] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)400worker2 # [ 0.034000] On node 0, zone DMA32: 40 pages in unavailable ranges401client # [ 0.036633] CPU topo: Num. cores per package: 1402worker2 # [ 0.034474] ACPI: PM-Timer IO Port: 0x608403lb # [ 0.002897] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)404client # [ 0.036633] CPU topo: Num. threads per package: 1405worker1 # [ 0.002933] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]406worker2 # [ 0.034486] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])407lb # [ 0.002904] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)408client # [ 0.036633] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs409worker1 # [ 0.002934] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]410lb # [ 0.002906] ACPI: FACS 0x000000005FFE0000 000040411worker2 # [ 0.034516] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23412client # [ 0.036657] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()413worker1 # [ 0.002935] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]414worker2 # [ 0.034519] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)415lb # [ 0.002907] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)416client # [ 0.036690] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]417worker1 # [ 0.002935] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]418worker2 # [ 0.034521] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)419lb # [ 0.002909] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)420client # [ 0.036692] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]421worker1 # [ 0.002936] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]422worker2 # [ 0.034522] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)423client # [ 0.036693] [mem 0x60000000-0xafffffff] available for PCI devices424worker1 # [ 0.002936] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]425lb # [ 0.002910] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)426worker2 # [ 0.034523] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)427client # [ 0.036695] Booting paravirtualized kernel on KVM428worker1 # [ 0.002937] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]429worker2 # [ 0.034524] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)430lb # [ 0.002912] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)431worker1 # [ 0.003162] No NUMA configuration found432client # [ 0.036699] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns433worker2 # [ 0.034526] ACPI: Using ACPI (MADT) for SMP configuration information434lb # [ 0.002913] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]435worker1 # [ 0.003163] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]436worker2 # [ 0.034527] ACPI: HPET id: 0x8086a201 base: 0xfed00000437client # [ 0.041197] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1438lb # [ 0.002914] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]439worker2 # [ 0.034531] TSC deadline timer available440worker1 # [ 0.003166] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]441worker1 # [ 0.005645] Zone ranges:442client # [ 0.043401] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152443worker2 # [ 0.034535] CPU topo: Max. logical packages: 1444lb # [ 0.002915] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]445worker1 # [ 0.005646] DMA [mem 0x0000000000001000-0x0000000000ffffff]446worker2 # [ 0.034536] CPU topo: Max. logical dies: 1447client # [ 0.043445] kvm-guest: PV spinlocks disabled, single CPU448lb # [ 0.002915] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]449worker2 # [ 0.034536] CPU topo: Max. dies per package: 1450worker1 # [ 0.005647] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]451worker2 # [ 0.034539] CPU topo: Max. threads per core: 1452worker1 # [ 0.005649] Normal empty453worker1 # [ 0.005649] Device empty454lb # [ 0.002916] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]455worker2 # [ 0.034540] CPU topo: Num. cores per package: 1456worker1 # [ 0.005650] Movable zone start for each node457lb # [ 0.002916] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]458worker2 # [ 0.034540] CPU topo: Num. threads per package: 1459worker1 # [ 0.005651] Early memory node ranges460lb # [ 0.002916] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]461worker1 # [ 0.005651] node 0: [mem 0x0000000000001000-0x000000000009efff]462worker2 # [ 0.034541] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs463lb # [ 0.003156] No NUMA configuration found464worker1 # [ 0.005652] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]465worker2 # [ 0.034559] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()466client # [ 0.043446] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/sj30989cgkidyy8h42lc8n59b9vy41qh-nixos-system-client-test/init regInfo=/nix/store/j0bs04qrv7kdcmc859n70b50kc7sz3gd-closure-info/registration console=ttyS0,115200n8 console=tty0467lb # [ 0.003157] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]468worker1 # [ 0.005653] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]469worker2 # [ 0.034591] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]470lb # [ 0.003160] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]471lb # [ 0.005583] Zone ranges:472worker1 # [ 0.005677] On node 0, zone DMA: 1 pages in unavailable ranges473client # [ 0.043541] Unknown kernel command line parameters "regInfo=/nix/store/j0bs04qrv7kdcmc859n70b50kc7sz3gd-closure-info/registration", will be passed to user space.474worker2 # [ 0.034592] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]475client # [ 0.043554] random: crng init done476lb # [ 0.005584] DMA [mem 0x0000000000001000-0x0000000000ffffff]477worker1 # [ 0.005960] On node 0, zone DMA: 97 pages in unavailable ranges478worker2 # [ 0.034594] [mem 0x60000000-0xafffffff] available for PCI devices479worker1 # [ 0.033485] On node 0, zone DMA32: 40 pages in unavailable ranges480lb # [ 0.005585] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]481worker2 # [ 0.034595] Booting paravirtualized kernel on KVM482client # [ 0.043555] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes483lb # [ 0.005586] Normal empty484worker1 # [ 0.033951] ACPI: PM-Timer IO Port: 0x608485lb # [ 0.005587] Device empty486client # [ 0.045708] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)487worker1 # [ 0.033964] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])488lb # [ 0.005588] Movable zone start for each node489worker2 # [ 0.034598] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns490lb # [ 0.005589] Early memory node ranges491client # [ 0.047072] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)492worker1 # [ 0.033992] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23493worker2 # [ 0.039095] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1494lb # [ 0.005589] node 0: [mem 0x0000000000001000-0x000000000009efff]495client # [ 0.047107] Fallback order for Node 0: 0496worker1 # [ 0.033995] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)497lb # [ 0.005590] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]498worker2 # [ 0.041366] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152499client # [ 0.047110] Built 1 zonelists, mobility grouping on. Total pages: 393078500worker1 # [ 0.033996] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)501client # [ 0.047111] Policy zone: DMA32502worker2 # [ 0.041411] kvm-guest: PV spinlocks disabled, single CPU503lb # [ 0.005591] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]504worker1 # [ 0.033997] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)505client # [ 0.049628] mem auto-init: stack:all(zero), heap alloc:on, heap free:off506lb # [ 0.005611] On node 0, zone DMA: 1 pages in unavailable ranges507worker1 # [ 0.033998] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)508client # [ 0.052890] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1509lb # [ 0.005887] On node 0, zone DMA: 97 pages in unavailable ranges510client # [ 0.056249] allocated 3145728 bytes of page_ext511worker1 # [ 0.033999] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)512lb # [ 0.034103] On node 0, zone DMA32: 40 pages in unavailable ranges513client # [ 0.066186] ftrace: allocating 48787 entries in 192 pages514lb # [ 0.034571] ACPI: PM-Timer IO Port: 0x608515worker1 # [ 0.034002] ACPI: Using ACPI (MADT) for SMP configuration information516client # [ 0.066188] ftrace: allocated 192 pages with 2 groups517lb # [ 0.034583] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])518client # [ 0.067026] Dynamic Preempt: lazy519worker1 # [ 0.034003] ACPI: HPET id: 0x8086a201 base: 0xfed00000520worker2 # [ 0.041413] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/lvqq63fax2ya6fsm3d7s7hfg0rmfbnl4-nixos-system-worker2-test/init regInfo=/nix/store/6pd0cf3aa37mjz7663lg8n7pgkfbwknl-closure-info/registration console=ttyS0,115200n8 console=tty0521worker1 # [ 0.034007] TSC deadline timer available522client # [ 0.067161] rcu: Preemptible hierarchical RCU implementation.523lb # [ 0.034611] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23524worker1 # [ 0.034011] CPU topo: Max. logical packages: 1525client # [ 0.067162] rcu: RCU event tracing is enabled.526lb # [ 0.034613] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)527worker1 # [ 0.034011] CPU topo: Max. logical dies: 1528worker2 # [ 0.041509] Unknown kernel command line parameters "regInfo=/nix/store/6pd0cf3aa37mjz7663lg8n7pgkfbwknl-closure-info/registration", will be passed to user space.529client # [ 0.067162] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.530worker1 # [ 0.034012] CPU topo: Max. dies per package: 1531worker2 # [ 0.041522] random: crng init done532lb # [ 0.034615] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)533client # [ 0.067164] Trampoline variant of Tasks RCU enabled.534worker1 # [ 0.034015] CPU topo: Max. threads per core: 1535lb # [ 0.034616] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)536worker2 # [ 0.041522] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes537worker1 # [ 0.034016] CPU topo: Num. cores per package: 1538client # [ 0.067164] Rude variant of Tasks RCU enabled.539client # [ 0.067165] Tracing variant of Tasks RCU enabled.540worker1 # [ 0.034016] CPU topo: Num. threads per package: 1541lb # [ 0.034617] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)542worker2 # [ 0.043771] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)543worker1 # [ 0.034016] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs544lb # [ 0.034618] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)545client # [ 0.067165] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.546worker2 # [ 0.045169] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)547lb # [ 0.034620] ACPI: Using ACPI (MADT) for SMP configuration information548worker1 # [ 0.034037] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()549client # [ 0.067166] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1550worker2 # [ 0.045202] Fallback order for Node 0: 0551lb # [ 0.034621] ACPI: HPET id: 0x8086a201 base: 0xfed00000552worker1 # [ 0.034069] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]553worker2 # [ 0.045205] Built 1 zonelists, mobility grouping on. Total pages: 393078554lb # [ 0.034625] TSC deadline timer available555worker2 # [ 0.045206] Policy zone: DMA32556client # [ 0.067182] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.557lb # [ 0.034629] CPU topo: Max. logical packages: 1558worker1 # [ 0.034071] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]559worker2 # [ 0.047807] mem auto-init: stack:all(zero), heap alloc:on, heap free:off560lb # [ 0.034629] CPU topo: Max. logical dies: 1561client # [ 0.067183] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.562worker1 # [ 0.034072] [mem 0x60000000-0xafffffff] available for PCI devices563lb # [ 0.034630] CPU topo: Max. dies per package: 1564worker2 # [ 0.051193] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1565worker1 # [ 0.034074] Booting paravirtualized kernel on KVM566lb # [ 0.034633] CPU topo: Max. threads per core: 1567client # [ 0.067184] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.568worker2 # [ 0.054682] allocated 3145728 bytes of page_ext569lb # [ 0.034634] CPU topo: Num. cores per package: 1570worker2 # [ 0.064716] ftrace: allocating 48787 entries in 192 pages571client # [ 0.071907] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16572worker1 # [ 0.034077] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns573lb # [ 0.034634] CPU topo: Num. threads per package: 1574worker2 # [ 0.064718] ftrace: allocated 192 pages with 2 groups575client # [ 0.072198] rcu: srcu_init: Setting srcu_struct sizes based on contention.576worker2 # [ 0.065606] Dynamic Preempt: lazy577worker1 # [ 0.038559] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1578lb # [ 0.034634] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs579worker2 # [ 0.065739] rcu: Preemptible hierarchical RCU implementation.580worker1 # [ 0.041044] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152581client # [ 0.072204] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns582lb # [ 0.034654] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()583worker2 # [ 0.065740] rcu: RCU event tracing is enabled.584worker1 # [ 0.041087] kvm-guest: PV spinlocks disabled, single CPU585lb # [ 0.034684] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]586worker2 # [ 0.065741] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.587client # [ 0.072308] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)588client # [ 0.075872] Console: colour VGA+ 80x25589worker2 # [ 0.065742] Trampoline variant of Tasks RCU enabled.590lb # [ 0.034686] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]591client # [ 0.075875] printk: legacy console [tty0] enabled592worker2 # [ 0.065743] Rude variant of Tasks RCU enabled.593lb # [ 0.034687] [mem 0x60000000-0xafffffff] available for PCI devices594worker2 # [ 0.065743] Tracing variant of Tasks RCU enabled.595client # [ 0.105514] printk: legacy console [ttyS0] enabled596lb # [ 0.034688] Booting paravirtualized kernel on KVM597client # [ 0.203480] ACPI: Core revision 20250807598worker2 # [ 0.065744] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.599lb # [ 0.034691] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns600worker1 # [ 0.041089] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/dn8gqamk2zyr2nvpgiwfbpgwzjyrjlpw-nixos-system-worker1-test/init regInfo=/nix/store/ndzr4x0h49369zkyn79cnanic26lxrvk-closure-info/registration console=ttyS0,115200n8 console=tty0601worker2 # [ 0.065744] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1602lb # [ 0.039177] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1603client # [ 0.204320] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns604worker2 # [ 0.065764] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.605lb # [ 0.041749] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152606worker1 # [ 0.041189] Unknown kernel command line parameters "regInfo=/nix/store/ndzr4x0h49369zkyn79cnanic26lxrvk-closure-info/registration", will be passed to user space.607client # [ 0.205839] APIC: Switch to symmetric I/O mode setup608worker1 # [ 0.041202] random: crng init done609lb # [ 0.041796] kvm-guest: PV spinlocks disabled, single CPU610worker2 # [ 0.065766] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.611client # [ 0.206795] x2apic enabled612worker1 # [ 0.041203] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes613worker2 # [ 0.065767] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.614client # [ 0.207557] APIC: Switched APIC routing to: physical x2apic615worker1 # [ 0.043369] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)616worker2 # [ 0.070567] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16617worker1 # [ 0.044448] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)618worker2 # [ 0.070846] rcu: srcu_init: Setting srcu_struct sizes based on contention.619worker1 # [ 0.044480] Fallback order for Node 0: 0620client # [ 0.209332] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1621lb # [ 0.041798] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/qfr7fghmwm537hb3f4f5is3yb8phm6jx-nixos-system-lb-test/init regInfo=/nix/store/s1480xaqc00gh7hz31b4z2fwjpnxi3sa-closure-info/registration console=ttyS0,115200n8 console=tty0622worker2 # [ 0.070854] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns623worker1 # [ 0.044483] Built 1 zonelists, mobility grouping on. Total pages: 393078624worker1 # [ 0.044484] Policy zone: DMA32625worker2 # [ 0.070964] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)626client # [ 0.210301] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns627worker1 # [ 0.047041] mem auto-init: stack:all(zero), heap alloc:on, heap free:off628lb # [ 0.041894] Unknown kernel command line parameters "regInfo=/nix/store/s1480xaqc00gh7hz31b4z2fwjpnxi3sa-closure-info/registration", will be passed to user space.629worker2 # [ 0.074517] Console: colour VGA+ 80x25630lb # [ 0.041906] random: crng init done631worker1 # [ 0.050234] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1632worker2 # [ 0.074520] printk: legacy console [tty0] enabled633client # [ 0.211936] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)634worker1 # [ 0.053904] allocated 3145728 bytes of page_ext635worker2 # [ 0.104265] printk: legacy console [ttyS0] enabled636lb # [ 0.041907] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes637worker1 # [ 0.063703] ftrace: allocating 48787 entries in 192 pages638worker2 # [ 0.200691] ACPI: Core revision 20250807639client # [ 0.213021] x86/cpu: User Mode Instruction Prevention (UMIP) activated640lb # [ 0.044151] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)641worker1 # [ 0.063705] ftrace: allocated 192 pages with 2 groups642worker1 # [ 0.064559] Dynamic Preempt: lazy643lb # [ 0.045336] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)644client # [ 0.214071] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127645worker1 # [ 0.064678] rcu: Preemptible hierarchical RCU implementation.646worker2 # [ 0.201559] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns647lb # [ 0.045368] Fallback order for Node 0: 0648worker1 # [ 0.064679] rcu: RCU event tracing is enabled.649client # [ 0.214934] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0650lb # [ 0.045371] Built 1 zonelists, mobility grouping on. Total pages: 393078651worker2 # [ 0.203059] APIC: Switch to symmetric I/O mode setup652worker1 # [ 0.064679] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.653lb # [ 0.045372] Policy zone: DMA32654worker2 # [ 0.204049] x2apic enabled655worker1 # [ 0.064681] Trampoline variant of Tasks RCU enabled.656lb # [ 0.047994] mem auto-init: stack:all(zero), heap alloc:on, heap free:off657worker1 # [ 0.064681] Rude variant of Tasks RCU enabled.658client # [ 0.215937] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto659worker2 # [ 0.204815] APIC: Switched APIC routing to: physical x2apic660worker1 # [ 0.064681] Tracing variant of Tasks RCU enabled.661lb # [ 0.051318] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1662lb # [ 0.055211] allocated 3145728 bytes of page_ext663worker1 # [ 0.064682] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.664client # [ 0.216934] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl665lb # [ 0.065240] ftrace: allocating 48787 entries in 192 pages666worker1 # [ 0.064683] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1667lb # [ 0.065242] ftrace: allocated 192 pages with 2 groups668client # [ 0.217934] Transient Scheduler Attacks: Vulnerable: No microcode669worker2 # [ 0.206636] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1670lb # [ 0.066126] Dynamic Preempt: lazy671worker1 # [ 0.064742] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.672lb # [ 0.066309] rcu: Preemptible hierarchical RCU implementation.673client # [ 0.218933] Spectre V2 : Mitigation: Enhanced / Automatic IBRS674lb # [ 0.066310] rcu: RCU event tracing is enabled.675worker2 # [ 0.207605] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns676worker1 # [ 0.064744] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.677client # [ 0.219934] Speculative Return Stack Overflow: Mitigation: Safe RET678lb # [ 0.066311] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.679lb # [ 0.066312] Trampoline variant of Tasks RCU enabled.680worker1 # [ 0.064744] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.681worker2 # [ 0.209242] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)682lb # [ 0.066313] Rude variant of Tasks RCU enabled.683worker1 # [ 0.069154] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16684lb # [ 0.066313] Tracing variant of Tasks RCU enabled.685worker2 # [ 0.210328] x86/cpu: User Mode Instruction Prevention (UMIP) activated686worker1 # [ 0.069433] rcu: srcu_init: Setting srcu_struct sizes based on contention.687lb # [ 0.066314] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.688worker2 # [ 0.211374] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127689lb # [ 0.066315] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1690worker1 # [ 0.069440] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns691client # [ 0.220933] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization692worker2 # [ 0.212239] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0693lb # [ 0.066331] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.694client # [ 0.222941] Spectre V2 : Enabling IBPB for BPF695worker1 # [ 0.069540] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)696worker1 # [ 0.073061] Console: colour VGA+ 80x25697lb # [ 0.066332] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.698worker1 # [ 0.073064] printk: legacy console [tty0] enabled699worker2 # [ 0.213243] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto700client # [ 0.223934] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier701worker1 # [ 0.102629] printk: legacy console [ttyS0] enabled702lb # [ 0.066333] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.703client # [ 0.224934] active return thunk: srso_alias_return_thunk704worker1 # [ 0.198053] ACPI: Core revision 20250807705worker2 # [ 0.215239] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl706lb # [ 0.070801] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16707client # [ 0.225955] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'708lb # [ 0.071083] rcu: srcu_init: Setting srcu_struct sizes based on contention.709worker2 # [ 0.216239] Transient Scheduler Attacks: Vulnerable: No microcode710worker1 # [ 0.198921] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns711client # [ 0.226933] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'712worker2 # [ 0.217238] Spectre V2 : Mitigation: Enhanced / Automatic IBRS713lb # [ 0.071089] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns714worker1 # [ 0.200409] APIC: Switch to symmetric I/O mode setup715client # [ 0.227933] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'716worker1 # [ 0.201338] x2apic enabled717worker2 # [ 0.218238] Speculative Return Stack Overflow: Mitigation: Safe RET718lb # [ 0.071201] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)719lb # [ 0.074728] Console: colour VGA+ 80x25720client # [ 0.228933] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'721worker1 # [ 0.202063] APIC: Switched APIC routing to: physical x2apic722lb # [ 0.074731] printk: legacy console [tty0] enabled723lb # [ 0.104204] printk: legacy console [ttyS0] enabled724client # [ 0.229934] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'725lb # [ 0.198905] ACPI: Core revision 20250807726worker1 # [ 0.203851] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1727client # [ 0.230934] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'728worker2 # [ 0.219238] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization729lb # [ 0.199809] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns730client # [ 0.231933] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'731worker1 # [ 0.204821] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns732worker2 # [ 0.221246] Spectre V2 : Enabling IBPB for BPF733lb # [ 0.201296] APIC: Switch to symmetric I/O mode setup734client # [ 0.232934] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256735lb # [ 0.202228] x2apic enabled736worker1 # [ 0.206455] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)737worker2 # [ 0.222239] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier738client # [ 0.234643] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64739lb # [ 0.202963] APIC: Switched APIC routing to: physical x2apic740worker2 # [ 0.223239] active return thunk: srso_alias_return_thunk741worker1 # [ 0.207538] x86/cpu: User Mode Instruction Prevention (UMIP) activated742client # [ 0.235632] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512743client # [ 0.235934] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024744worker1 # [ 0.208570] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127745worker2 # [ 0.224259] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'746lb # [ 0.204766] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1747client # [ 0.236933] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8748worker1 # [ 0.209452] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0749worker2 # [ 0.225238] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'750worker2 # [ 0.226238] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'751lb # [ 0.205744] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns752client # [ 0.237933] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.753worker1 # [ 0.210456] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto754worker2 # [ 0.227238] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'755lb # [ 0.207376] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)756worker1 # [ 0.211452] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl757worker2 # [ 0.228238] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'758lb # [ 0.208463] x86/cpu: User Mode Instruction Prevention (UMIP) activated759worker1 # [ 0.212452] Transient Scheduler Attacks: Vulnerable: No microcode760worker2 # [ 0.229238] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'761lb # [ 0.209544] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127762worker1 # [ 0.213452] Spectre V2 : Mitigation: Enhanced / Automatic IBRS763worker2 # [ 0.230238] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'764lb # [ 0.210373] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0765worker1 # [ 0.214452] Speculative Return Stack Overflow: Mitigation: Safe RET766worker2 # [ 0.232239] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256767worker2 # [ 0.233238] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64768lb # [ 0.211377] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto769worker2 # [ 0.234238] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512770lb # [ 0.213373] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl771worker2 # [ 0.235239] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024772worker1 # [ 0.215452] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization773lb # [ 0.214374] Transient Scheduler Attacks: Vulnerable: No microcode774worker2 # [ 0.236238] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8775worker1 # [ 0.217460] Spectre V2 : Enabling IBPB for BPF776lb # [ 0.215373] Spectre V2 : Mitigation: Enhanced / Automatic IBRS777lb # [ 0.216373] Speculative Return Stack Overflow: Mitigation: Safe RET778worker1 # [ 0.218453] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier779worker2 # [ 0.237238] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.780worker1 # [ 0.219453] active return thunk: srso_alias_return_thunk781worker1 # [ 0.220473] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'782worker1 # [ 0.221452] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'783lb # [ 0.217373] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization784worker1 # [ 0.222452] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'785lb # [ 0.219380] Spectre V2 : Enabling IBPB for BPF786worker1 # [ 0.223452] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'787lb # [ 0.220374] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier788worker1 # [ 0.224452] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'789lb # [ 0.221374] active return thunk: srso_alias_return_thunk790worker1 # [ 0.225452] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'791lb # [ 0.222395] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'792worker1 # [ 0.226452] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'793lb # [ 0.223373] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'794worker1 # [ 0.227452] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256795lb # [ 0.224373] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'796worker1 # [ 0.228452] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64797lb # [ 0.225373] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'798worker1 # [ 0.229452] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512799lb # [ 0.226373] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'800worker1 # [ 0.230452] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024801lb # [ 0.227373] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'802worker1 # [ 0.231452] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8803lb # [ 0.228373] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'804worker1 # [ 0.232452] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.805lb # [ 0.229373] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256806lb # [ 0.231083] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64807lb # [ 0.232087] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512808lb # [ 0.232373] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024809lb # [ 0.233373] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8810lb # [ 0.234373] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.811client # [ 0.272354] Freeing SMP alternatives memory: 44K812client # [ 0.272936] pid_max: default: 32768 minimum: 301813client # [ 0.274018] LSM: initializing lsm=capability,landlock,yama,bpf,ima814client # [ 0.275038] landlock: Up and running.815client # [ 0.275683] Yama: becoming mindful.816client # [ 0.276593] LSM support for eBPF active817client # [ 0.277063] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)818client # [ 0.277978] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)819client # [ 0.281206] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)820worker2 # [ 0.272244] Freeing SMP alternatives memory: 44K821client # [ 0.282550] Performance Events: Fam17h+ core perfctr, AMD PMU driver.822worker2 # [ 0.273039] pid_max: default: 32768 minimum: 301823client # [ 0.282942] ... version: 2824worker2 # [ 0.273900] LSM: initializing lsm=capability,landlock,yama,bpf,ima825client # [ 0.283682] ... bit width: 48826worker2 # [ 0.274336] landlock: Up and running.827client # [ 0.283936] ... generic counters: 6828worker2 # [ 0.275673] Yama: becoming mindful.829client # [ 0.284643] ... generic bitmap: 000000000000003f830worker2 # [ 0.276450] LSM support for eBPF active831client # [ 0.284936] ... fixed-purpose counters: 0832client # [ 0.285660] ... fixed-purpose bitmap: 0000000000000000833worker2 # [ 0.277372] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)834client # [ 0.285936] ... value mask: 0000ffffffffffff835worker2 # [ 0.278283] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)836client # [ 0.286861] ... max period: 00007fffffffffff837client # [ 0.287685] ... global_ctrl mask: 000000000000003f838client # [ 0.288032] signal: max sigframe size: 3376839worker2 # [ 0.281136] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)840client # [ 0.288791] rcu: Hierarchical SRCU implementation.841client # [ 0.289492] rcu: Max phase no-delay instances is 400.842worker2 # [ 0.281817] Performance Events: Fam17h+ core perfctr, AMD PMU driver.843worker2 # [ 0.282243] ... version: 2844worker2 # [ 0.282974] ... bit width: 48845worker2 # [ 0.283241] ... generic counters: 6846worker1 # [ 0.267285] Freeing SMP alternatives memory: 44K847worker2 # [ 0.284009] ... generic bitmap: 000000000000003f848worker1 # [ 0.267997] pid_max: default: 32768 minimum: 301849worker2 # [ 0.284253] ... fixed-purpose counters: 0850worker1 # [ 0.268536] LSM: initializing lsm=capability,landlock,yama,bpf,ima851worker2 # [ 0.285004] ... fixed-purpose bitmap: 0000000000000000852client # [ 0.294625] smp: Bringing up secondary CPUs ...853worker1 # [ 0.269554] landlock: Up and running.854client # [ 0.294950] smp: Brought up 1 node, 1 CPU855worker2 # [ 0.285241] ... value mask: 0000ffffffffffff856worker1 # [ 0.270452] Yama: becoming mindful.857client # [ 0.295589] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)858worker2 # [ 0.286097] ... max period: 00007fffffffffff859worker1 # [ 0.271239] LSM support for eBPF active860worker2 # [ 0.286889] ... global_ctrl mask: 000000000000003f861worker1 # [ 0.272019] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)862worker2 # [ 0.287365] signal: max sigframe size: 3376863client # [ 0.296083] Memory: 1454600K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109308K reserved, 0K cma-reserved)864lb # [ 0.268996] Freeing SMP alternatives memory: 44K865worker1 # [ 0.273328] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)866worker2 # [ 0.288191] rcu: Hierarchical SRCU implementation.867client # [ 0.297185] devtmpfs: initialized868lb # [ 0.269376] pid_max: default: 32768 minimum: 301869worker2 # [ 0.288837] rcu: Max phase no-delay instances is 400.870client # [ 0.297896] x86/mm: Memory block size: 128MB871lb # [ 0.270473] LSM: initializing lsm=capability,landlock,yama,bpf,ima872worker1 # [ 0.275612] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)873lb # [ 0.271478] landlock: Up and running.874lb # [ 0.272373] Yama: becoming mindful.875client # [ 0.299470] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)876lb # [ 0.273174] LSM support for eBPF active877worker1 # [ 0.277018] Performance Events: Fam17h+ core perfctr, AMD PMU driver.878worker1 # [ 0.277461] ... version: 2879client # [ 0.299960] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).880lb # [ 0.273950] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)881worker1 # [ 0.278136] ... bit width: 48882client # [ 0.301028] pinctrl core: initialized pinctrl subsystem883worker1 # [ 0.278454] ... generic counters: 6884worker2 # [ 0.293794] smp: Bringing up secondary CPUs ...885lb # [ 0.274413] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)886client # [ 0.302132] PM: RTC time: 13:45:14, date: 2026-09-22887worker2 # [ 0.294254] smp: Brought up 1 node, 1 CPU888worker1 # [ 0.279160] ... generic bitmap: 000000000000003f889worker1 # [ 0.279454] ... fixed-purpose counters: 0890worker2 # [ 0.294956] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)891worker1 # [ 0.280128] ... fixed-purpose bitmap: 0000000000000000892lb # [ 0.277636] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)893worker1 # [ 0.280454] ... value mask: 0000ffffffffffff894worker1 # [ 0.281327] ... max period: 00007fffffffffff895worker2 # [ 0.295448] Memory: 1454600K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109312K reserved, 0K cma-reserved)896client # [ 0.305773] NET: Registered PF_NETLINK/PF_ROUTE protocol family897lb # [ 0.278962] Performance Events: Fam17h+ core perfctr, AMD PMU driver.898worker1 # [ 0.282109] ... global_ctrl mask: 000000000000003f899worker2 # [ 0.296516] devtmpfs: initialized900lb # [ 0.279382] ... version: 2901worker1 # [ 0.282549] signal: max sigframe size: 3376902worker2 # [ 0.297296] x86/mm: Memory block size: 128MB903client # [ 0.307108] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations904lb # [ 0.280105] ... bit width: 48905worker1 # [ 0.283369] rcu: Hierarchical SRCU implementation.906lb # [ 0.280375] ... generic counters: 6907client # [ 0.307957] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations908worker1 # [ 0.284009] rcu: Max phase no-delay instances is 400.909lb # [ 0.281085] ... generic bitmap: 000000000000003f910worker2 # [ 0.299093] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)911lb # [ 0.281376] ... fixed-purpose counters: 0912client # [ 0.309226] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations913lb # [ 0.282102] ... fixed-purpose bitmap: 0000000000000000914client # [ 0.309946] audit: initializing netlink subsys (disabled)915worker2 # [ 0.300145] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).916lb # [ 0.282375] ... value mask: 0000ffffffffffff917worker2 # [ 0.301334] pinctrl core: initialized pinctrl subsystem918client # [ 0.311062] thermal_sys: Registered thermal governor 'fair_share'919lb # [ 0.283258] ... max period: 00007fffffffffff920worker2 # [ 0.302430] PM: RTC time: 13:45:14, date: 2026-09-22921client # [ 0.311065] thermal_sys: Registered thermal governor 'bang_bang'922lb # [ 0.284050] ... global_ctrl mask: 000000000000003f923client # [ 0.311937] thermal_sys: Registered thermal governor 'step_wise'924worker1 # [ 0.288999] smp: Bringing up secondary CPUs ...925lb # [ 0.284472] signal: max sigframe size: 3376926worker1 # [ 0.289469] smp: Brought up 1 node, 1 CPU927client # [ 0.312903] thermal_sys: Registered thermal governor 'user_space'928lb # [ 0.285299] rcu: Hierarchical SRCU implementation.929worker1 # [ 0.290119] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)930lb # [ 0.285997] rcu: Max phase no-delay instances is 400.931client # [ 0.313651] audit: type=2000 audit(1790084715.635:1): state=initialized audit_enabled=0 res=1932worker2 # [ 0.306196] NET: Registered PF_NETLINK/PF_ROUTE protocol family933client # [ 0.314939] thermal_sys: Registered thermal governor 'power_allocator'934client # [ 0.314953] cpuidle: using governor menu935worker1 # [ 0.290629] Memory: 1454604K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109308K reserved, 0K cma-reserved)936worker2 # [ 0.307532] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations937worker1 # [ 0.291697] devtmpfs: initialized938client # [ 0.317131] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5939worker1 # [ 0.292460] x86/mm: Memory block size: 128MB940worker2 # [ 0.308266] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations941client # [ 0.318234] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]942lb # [ 0.291146] smp: Bringing up secondary CPUs ...943worker2 # [ 0.309532] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations944worker1 # [ 0.294176] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)945client # [ 0.318939] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry946lb # [ 0.291390] smp: Brought up 1 node, 1 CPU947worker2 # [ 0.310251] audit: initializing netlink subsys (disabled)948client # [ 0.319948] PCI: Using configuration type 1 for base access949lb # [ 0.292036] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)950worker1 # [ 0.295285] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).951worker2 # [ 0.311409] thermal_sys: Registered thermal governor 'fair_share'952worker1 # [ 0.296524] pinctrl core: initialized pinctrl subsystem953client # [ 0.321006] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.954worker2 # [ 0.311411] thermal_sys: Registered thermal governor 'bang_bang'955worker1 # [ 0.297647] PM: RTC time: 13:45:14, date: 2026-09-22956lb # [ 0.292514] Memory: 1454588K/1572312K available (17247K kernel code, 2727K rwdata, 13616K rodata, 3652K init, 2976K bss, 109316K reserved, 0K cma-reserved)957worker2 # [ 0.312242] thermal_sys: Registered thermal governor 'step_wise'958lb # [ 0.293642] devtmpfs: initialized959worker2 # [ 0.313137] thermal_sys: Registered thermal governor 'user_space'960lb # [ 0.294443] x86/mm: Memory block size: 128MB961worker2 # [ 0.313935] audit: type=2000 audit(1790084715.650:1): state=initialized audit_enabled=0 res=1962worker2 # [ 0.315244] thermal_sys: Registered thermal governor 'power_allocator'963worker1 # [ 0.301504] NET: Registered PF_NETLINK/PF_ROUTE protocol family964lb # [ 0.296295] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)965worker2 # [ 0.315258] cpuidle: using governor menu966client # [ 0.326213] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages967client # [ 0.326937] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page968lb # [ 0.297241] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).969worker1 # [ 0.302964] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations970worker2 # [ 0.317449] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5971lb # [ 0.298457] pinctrl core: initialized pinctrl subsystem972worker1 # [ 0.303475] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations973worker2 # [ 0.318481] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]974lb # [ 0.299610] PM: RTC time: 13:45:14, date: 2026-09-22975worker1 # [ 0.304721] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations976worker2 # [ 0.319244] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry977worker1 # [ 0.305466] audit: initializing netlink subsys (disabled)978worker2 # [ 0.320253] PCI: Using configuration type 1 for base access979worker1 # [ 0.306574] thermal_sys: Registered thermal governor 'fair_share'980client # [ 0.327930] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages981worker2 # [ 0.321392] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.982worker1 # [ 0.306577] thermal_sys: Registered thermal governor 'bang_bang'983client # [ 0.332936] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page984lb # [ 0.303479] NET: Registered PF_NETLINK/PF_ROUTE protocol family985worker1 # [ 0.307455] thermal_sys: Registered thermal governor 'step_wise'986worker1 # [ 0.308378] thermal_sys: Registered thermal governor 'user_space'987lb # [ 0.304873] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations988worker1 # [ 0.309175] audit: type=2000 audit(1790084715.664:1): state=initialized audit_enabled=0 res=1989lb # [ 0.305401] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations990worker1 # [ 0.310457] thermal_sys: Registered thermal governor 'power_allocator'991worker1 # [ 0.310480] cpuidle: using governor menu992lb # [ 0.306646] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations993lb # [ 0.307393] audit: initializing netlink subsys (disabled)994worker1 # [ 0.312633] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5995worker2 # [ 0.328490] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages996lb # [ 0.308577] thermal_sys: Registered thermal governor 'fair_share'997worker1 # [ 0.313685] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]998lb # [ 0.308579] thermal_sys: Registered thermal governor 'bang_bang'999worker2 # [ 0.329242] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1000worker1 # [ 0.314458] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry1001lb # [ 0.309377] thermal_sys: Registered thermal governor 'step_wise'1002worker1 # [ 0.315467] PCI: Using configuration type 1 for base access1003lb # [ 0.310326] thermal_sys: Registered thermal governor 'user_space'1004client # [ 0.343308] ACPI: Added _OSI(Module Device)1005client # [ 0.343938] ACPI: Added _OSI(Processor Device)1006worker1 # [ 0.316547] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.1007lb # [ 0.311172] audit: type=2000 audit(1790084715.673:1): state=initialized audit_enabled=0 res=11008client # [ 0.344652] ACPI: Added _OSI(Processor Aggregator Device)1009lb # [ 0.313378] thermal_sys: Registered thermal governor 'power_allocator'1010lb # [ 0.313404] cpuidle: using governor menu1011worker2 # [ 0.334242] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1012worker2 # [ 0.335242] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1013lb # [ 0.315623] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.51014lb # [ 0.316628] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]1015worker1 # [ 0.321720] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1016lb # [ 0.317379] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry1017worker1 # [ 0.322455] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1018lb # [ 0.318387] PCI: Using configuration type 1 for base access1019lb # [ 0.319523] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.1020client # [ 0.353480] ACPI: 1 ACPI AML tables successfully acquired and loaded1021worker2 # [ 0.343938] ACPI: Added _OSI(Module Device)1022worker1 # [ 0.327455] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1023client # [ 0.354856] ACPI: Interpreter enabled1024worker2 # [ 0.344243] ACPI: Added _OSI(Processor Device)1025client # [ 0.355380] ACPI: PM: (supports S0 S3 S4 S5)1026worker1 # [ 0.328455] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1027worker2 # [ 0.345012] ACPI: Added _OSI(Processor Aggregator Device)1028lb # [ 0.324494] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1029client # [ 0.357938] ACPI: Using IOAPIC for interrupt routing1030lb # [ 0.325376] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page1031client # [ 0.358826] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1032client # [ 0.361936] PCI: Using E820 reservations for host bridge windows1033lb # [ 0.330377] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1034client # [ 0.363056] ACPI: Enabled 2 GPEs in block 00 to 3F1035worker2 # [ 0.351830] ACPI: 1 ACPI AML tables successfully acquired and loaded1036lb # [ 0.331375] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page1037worker1 # [ 0.338847] ACPI: Added _OSI(Module Device)1038worker1 # [ 0.339457] ACPI: Added _OSI(Processor Device)1039worker1 # [ 0.340200] ACPI: Added _OSI(Processor Aggregator Device)1040worker2 # [ 0.355147] ACPI: Interpreter enabled1041worker2 # [ 0.355709] ACPI: PM: (supports S0 S3 S4 S5)1042worker2 # [ 0.356242] ACPI: Using IOAPIC for interrupt routing1043worker2 # [ 0.357154] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1044client # [ 0.371718] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1045lb # [ 0.341723] ACPI: Added _OSI(Module Device)1046client # [ 0.373383] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1047lb # [ 0.342377] ACPI: Added _OSI(Processor Device)1048worker2 # [ 0.362242] PCI: Using E820 reservations for host bridge windows1049client # [ 0.374060] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1050lb # [ 0.343210] ACPI: Added _OSI(Processor Aggregator Device)1051worker2 # [ 0.363378] ACPI: Enabled 2 GPEs in block 00 to 3F1052worker1 # [ 0.349004] ACPI: 1 ACPI AML tables successfully acquired and loaded1053client # [ 0.375058] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1054worker1 # [ 0.350400] ACPI: Interpreter enabled1055client # [ 0.376314] PCI host bridge to bus 0000:001056worker1 # [ 0.350927] ACPI: PM: (supports S0 S3 S4 S5)1057client # [ 0.376940] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1058client # [ 0.377937] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1059client # [ 0.378936] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1060worker1 # [ 0.353476] ACPI: Using IOAPIC for interrupt routing1061client # [ 0.379936] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1062worker1 # [ 0.354515] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1063client # [ 0.380937] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1064worker2 # [ 0.372052] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1065lb # [ 0.352426] ACPI: 1 ACPI AML tables successfully acquired and loaded1066client # [ 0.381936] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1067worker1 # [ 0.357454] PCI: Using E820 reservations for host bridge windows1068client # [ 0.382937] pci_bus 0000:00: root bus resource [bus 00-ff]1069worker2 # [ 0.373544] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1070worker1 # [ 0.358547] ACPI: Enabled 2 GPEs in block 00 to 3F1071client # [ 0.383914] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1072worker2 # [ 0.374328] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1073lb # [ 0.356287] ACPI: Interpreter enabled1074lb # [ 0.356839] ACPI: PM: (supports S0 S3 S4 S5)1075worker2 # [ 0.375363] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1076client # [ 0.385451] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1077lb # [ 0.357376] ACPI: Using IOAPIC for interrupt routing1078worker2 # [ 0.376621] PCI host bridge to bus 0000:001079lb # [ 0.358246] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1080worker2 # [ 0.377246] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1081worker2 # [ 0.378241] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1082worker2 # [ 0.379241] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1083worker2 # [ 0.380242] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1084worker1 # [ 0.367188] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1085lb # [ 0.363376] PCI: Using E820 reservations for host bridge windows1086worker2 # [ 0.381242] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1087lb # [ 0.364466] ACPI: Enabled 2 GPEs in block 00 to 3F1088worker2 # [ 0.382241] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1089worker1 # [ 0.368226] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1090client # [ 0.388022] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1091worker2 # [ 0.383241] pci_bus 0000:00: root bus resource [bus 00-ff]1092client # [ 0.388949] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1093worker1 # [ 0.369534] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1094worker2 # [ 0.384202] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1095worker1 # [ 0.370577] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1096client # [ 0.389922] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1097worker1 # [ 0.371801] PCI host bridge to bus 0000:001098worker2 # [ 0.385682] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1099client # [ 0.391171] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1100worker1 # [ 0.372458] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1101worker1 # [ 0.373454] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1102client # [ 0.392698] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1103worker1 # [ 0.374454] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1104lb # [ 0.372178] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1105worker1 # [ 0.375455] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1106lb # [ 0.373166] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1107worker1 # [ 0.376455] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1108client # [ 0.395972] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1109lb # [ 0.374452] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1110client # [ 0.396811] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1111worker1 # [ 0.377455] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1112lb # [ 0.375498] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1113client # [ 0.397758] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1114worker1 # [ 0.378461] pci_bus 0000:00: root bus resource [bus 00-ff]1115lb # [ 0.376751] PCI host bridge to bus 0000:001116client # [ 0.398736] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1117worker2 # [ 0.388395] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1118worker1 # [ 0.379468] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1119lb # [ 0.377380] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1120worker2 # [ 0.389332] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1121lb # [ 0.378377] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1122worker1 # [ 0.380919] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1123client # [ 0.400440] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1124worker2 # [ 0.390306] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1125lb # [ 0.379380] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1126worker2 # [ 0.391570] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1127lb # [ 0.380413] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1128client # [ 0.402971] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1129lb # [ 0.381376] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1130worker2 # [ 0.392958] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1131client # [ 0.403830] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1132lb # [ 0.382376] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1133client # [ 0.404782] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1134lb # [ 0.383376] pci_bus 0000:00: root bus resource [bus 00-ff]1135worker2 # [ 0.395279] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1136lb # [ 0.384377] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1137client # [ 0.406384] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1138worker2 # [ 0.396127] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1139lb # [ 0.385810] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1140worker1 # [ 0.383556] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1141worker2 # [ 0.397041] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1142worker1 # [ 0.384467] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1143worker2 # [ 0.398098] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1144worker1 # [ 0.385438] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1145client # [ 0.409975] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1146worker2 # [ 0.399823] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1147client # [ 0.410842] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1148worker1 # [ 0.386667] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1149client # [ 0.411692] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1150worker2 # [ 0.401250] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1151worker1 # [ 0.388159] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1152client # [ 0.412516] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1153worker2 # [ 0.402102] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1154worker2 # [ 0.402980] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1155lb # [ 0.388472] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1156worker1 # [ 0.391494] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1157client # [ 0.414882] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1158worker2 # [ 0.403796] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1159lb # [ 0.389389] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1160worker1 # [ 0.392358] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1161client # [ 0.415563] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1162lb # [ 0.390331] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1163worker1 # [ 0.393197] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1164client # [ 0.415958] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1165lb # [ 0.391554] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1166worker1 # [ 0.393461] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1167client # [ 0.416943] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1168worker2 # [ 0.406249] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1169worker1 # [ 0.395043] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1170lb # [ 0.393117] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1171worker2 # [ 0.407248] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1172client # [ 0.418507] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1173worker2 # [ 0.408185] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1174client # [ 0.419949] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1175worker1 # [ 0.397494] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1176lb # [ 0.396417] pci 0000:00:02.0: BAR 0 [io 0xc100-0xc11f]1177worker2 # [ 0.409664] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1178worker1 # [ 0.398287] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1179client # [ 0.420874] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1180lb # [ 0.397265] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1181worker1 # [ 0.399174] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1182lb # [ 0.398098] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1183worker2 # [ 0.411281] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1184client # [ 0.422345] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1185worker2 # [ 0.412138] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1186lb # [ 0.398382] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1187worker1 # [ 0.400032] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1188worker2 # [ 0.412999] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1189lb # [ 0.399965] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1190worker2 # [ 0.413247] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1191client # [ 0.424429] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1192client # [ 0.425735] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1193worker2 # [ 0.414853] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1194lb # [ 0.401416] pci 0000:00:03.0: BAR 0 [io 0xc120-0xc13f]1195worker1 # [ 0.403462] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1196lb # [ 0.402276] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1197worker1 # [ 0.404314] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1198client # [ 0.427443] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1199worker2 # [ 0.416109] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1200worker1 # [ 0.405156] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1201lb # [ 0.403103] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1202worker2 # [ 0.416989] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1203client # [ 0.428949] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1204worker1 # [ 0.406020] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1205lb # [ 0.404754] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1206client # [ 0.429913] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1207worker2 # [ 0.417845] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1208client # [ 0.431344] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1209worker1 # [ 0.408504] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1210worker2 # [ 0.419289] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1211lb # [ 0.407423] pci 0000:00:04.0: BAR 0 [io 0xc000-0xc07f]1212worker1 # [ 0.409388] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1213worker2 # [ 0.420263] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1214lb # [ 0.408280] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1215client # [ 0.433949] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1216lb # [ 0.409119] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1217worker1 # [ 0.410214] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1218worker2 # [ 0.421850] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1219client # [ 0.434887] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1220worker1 # [ 0.410461] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1221lb # [ 0.410743] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1222worker2 # [ 0.423022] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1223client # [ 0.436289] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1224worker1 # [ 0.412005] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1225worker2 # [ 0.424006] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1226lb # [ 0.412446] pci 0000:00:05.0: BAR 0 [io 0xc140-0xc15f]1227worker1 # [ 0.413467] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1228client # [ 0.437975] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1229worker2 # [ 0.425712] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1230lb # [ 0.413287] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1231worker1 # [ 0.414454] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1232client # [ 0.438839] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1233lb # [ 0.414069] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1234worker2 # [ 0.427290] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1235client # [ 0.439710] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1236worker1 # [ 0.416004] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1237lb # [ 0.414382] pci 0000:00:05.0: ROM [mem 0xfeb80000-0xfebbffff pref]1238worker2 # [ 0.428257] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1239lb # [ 0.415918] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1240client # [ 0.441336] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1241worker2 # [ 0.429775] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1242worker1 # [ 0.417906] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1243lb # [ 0.417389] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1244worker1 # [ 0.419251] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1245client # [ 0.444639] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1246worker2 # [ 0.431301] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1247lb # [ 0.418398] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1248client # [ 0.444943] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1249worker2 # [ 0.432153] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1250worker1 # [ 0.420911] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1251client # [ 0.445876] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1252lb # [ 0.419931] pci 0000:00:07.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1253worker2 # [ 0.433008] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1254client # [ 0.447324] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1255worker1 # [ 0.422467] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1256worker2 # [ 0.433785] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1257worker1 # [ 0.423454] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1258lb # [ 0.421389] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1259client # [ 0.448647] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1260lb # [ 0.422399] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1261worker1 # [ 0.425024] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1262worker2 # [ 0.435487] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1263client # [ 0.449165] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1264lb # [ 0.423981] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1265worker2 # [ 0.436248] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1266client # [ 0.450884] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1267worker2 # [ 0.437262] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1268worker1 # [ 0.427122] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1269lb # [ 0.425423] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1270worker1 # [ 0.427477] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1271client # [ 0.451798] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1272lb # [ 0.426398] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1273worker2 # [ 0.438836] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1274client # [ 0.453663] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1275worker1 # [ 0.429038] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1276lb # [ 0.427919] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1277worker2 # [ 0.439930] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1278worker2 # [ 0.440440] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1279lb # [ 0.429411] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1280worker1 # [ 0.431310] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1281worker2 # [ 0.441802] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1282worker1 # [ 0.432159] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1283lb # [ 0.430398] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1284worker1 # [ 0.432476] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1285worker2 # [ 0.442449] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1286lb # [ 0.431947] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1287worker2 # [ 0.443760] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1288worker1 # [ 0.434456] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1289lb # [ 0.433425] pci 0000:00:0a.0: BAR 0 [io 0xc080-0xc0bf]1290lb # [ 0.434339] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1291worker1 # [ 0.437153] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1292lb # [ 0.435179] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1293worker1 # [ 0.437462] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1294worker1 # [ 0.438454] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1295lb # [ 0.436836] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1296client # [ 0.454157] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1297worker1 # [ 0.440034] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1298client # [ 0.456698] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1299lb # [ 0.438431] pci 0000:00:0b.0: BAR 0 [io 0xc160-0xc17f]1300worker1 # [ 0.441195] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1301lb # [ 0.439306] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1302client # [ 0.457309] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1303lb # [ 0.440096] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1304client # [ 0.458228] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1305worker1 # [ 0.441656] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1306lb # [ 0.440950] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1307client # [ 0.459194] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1308worker2 # [ 0.444467] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1309worker1 # [ 0.444286] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1310lb # [ 0.442037] pci 0000:00:1d.0: BAR 4 [io 0xc180-0xc19f]1311worker2 # [ 0.446759] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1312client # [ 0.460965] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1313worker1 # [ 0.445291] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1314lb # [ 0.442573] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1315client # [ 0.461816] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1316worker2 # [ 0.447523] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1317worker1 # [ 0.447182] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1318lb # [ 0.444434] pci 0000:00:1d.1: BAR 4 [io 0xc1a0-0xc1bf]1319client # [ 0.463008] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1320worker2 # [ 0.448528] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1321lb # [ 0.445486] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1322worker2 # [ 0.449494] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1323client # [ 0.464573] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1324lb # [ 0.447039] pci 0000:00:1d.2: BAR 4 [io 0xc1c0-0xc1df]1325worker2 # [ 0.451083] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1326worker2 # [ 0.451964] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1327client # [ 0.468701] ACPI: PCI: Interrupt link LNKA configured for IRQ 101328worker2 # [ 0.452556] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1329client # [ 0.469048] ACPI: PCI: Interrupt link LNKB configured for IRQ 101330worker2 # [ 0.453781] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1331client # [ 0.470036] ACPI: PCI: Interrupt link LNKC configured for IRQ 111332client # [ 0.471037] ACPI: PCI: Interrupt link LNKD configured for IRQ 111333client # [ 0.472035] ACPI: PCI: Interrupt link LNKE configured for IRQ 101334client # [ 0.473033] ACPI: PCI: Interrupt link LNKF configured for IRQ 101335worker2 # [ 0.457337] ACPI: PCI: Interrupt link LNKA configured for IRQ 101336worker1 # [ 0.447698] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1337client # [ 0.474044] ACPI: PCI: Interrupt link LNKG configured for IRQ 111338worker2 # [ 0.458344] ACPI: PCI: Interrupt link LNKB configured for IRQ 101339client # [ 0.475045] ACPI: PCI: Interrupt link LNKH configured for IRQ 111340worker2 # [ 0.459342] ACPI: PCI: Interrupt link LNKC configured for IRQ 111341worker1 # [ 0.450265] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1342client # [ 0.475977] ACPI: PCI: Interrupt link GSIA configured for IRQ 161343worker2 # [ 0.460341] ACPI: PCI: Interrupt link LNKD configured for IRQ 111344lb # [ 0.447639] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1345worker1 # [ 0.451743] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1346client # [ 0.476951] ACPI: PCI: Interrupt link GSIB configured for IRQ 171347worker2 # [ 0.461342] ACPI: PCI: Interrupt link LNKE configured for IRQ 101348client # [ 0.477951] ACPI: PCI: Interrupt link GSIC configured for IRQ 181349lb # [ 0.449988] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1350worker2 # [ 0.462340] ACPI: PCI: Interrupt link LNKF configured for IRQ 101351worker1 # [ 0.452743] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1352client # [ 0.478920] ACPI: PCI: Interrupt link GSID configured for IRQ 191353worker2 # [ 0.463341] ACPI: PCI: Interrupt link LNKG configured for IRQ 111354lb # [ 0.450666] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1355client # [ 0.479720] ACPI: PCI: Interrupt link GSIE configured for IRQ 201356worker1 # [ 0.453743] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1357worker2 # [ 0.464336] ACPI: PCI: Interrupt link LNKH configured for IRQ 111358client # [ 0.480681] ACPI: PCI: Interrupt link GSIF configured for IRQ 211359lb # [ 0.451760] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1360worker2 # [ 0.465275] ACPI: PCI: Interrupt link GSIA configured for IRQ 161361client # [ 0.480948] ACPI: PCI: Interrupt link GSIG configured for IRQ 221362worker1 # [ 0.455483] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1363worker2 # [ 0.466252] ACPI: PCI: Interrupt link GSIB configured for IRQ 171364lb # [ 0.452639] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1365client # [ 0.481929] ACPI: PCI: Interrupt link GSIH configured for IRQ 231366worker1 # [ 0.456327] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1367worker2 # [ 0.467259] ACPI: PCI: Interrupt link GSIC configured for IRQ 181368worker2 # [ 0.468251] ACPI: PCI: Interrupt link GSID configured for IRQ 191369lb # [ 0.454404] pci 0000:00:1f.2: BAR 4 [io 0xc1e0-0xc1ff]1370client # [ 0.483770] iommu: Default domain type: Translated1371worker1 # [ 0.457513] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1372worker2 # [ 0.469248] ACPI: PCI: Interrupt link GSIE configured for IRQ 201373lb # [ 0.455245] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1374client # [ 0.484537] iommu: DMA domain TLB invalidation policy: lazy mode1375worker2 # [ 0.470192] ACPI: PCI: Interrupt link GSIF configured for IRQ 211376client # [ 0.485209] ACPI: bus type USB registered1377worker1 # [ 0.459911] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1378lb # [ 0.456403] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1379worker2 # [ 0.470998] ACPI: PCI: Interrupt link GSIG configured for IRQ 221380client # [ 0.485982] usbcore: registered new interface driver usbfs1381worker2 # [ 0.471254] ACPI: PCI: Interrupt link GSIH configured for IRQ 231382client # [ 0.486887] usbcore: registered new interface driver hub1383lb # [ 0.458030] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1384client # [ 0.487604] usbcore: registered new device driver usb1385worker2 # [ 0.473219] iommu: Default domain type: Translated1386client # [ 0.488984] NetLabel: Initializing1387worker1 # [ 0.463600] ACPI: PCI: Interrupt link LNKA configured for IRQ 101388worker2 # [ 0.473870] iommu: DMA domain TLB invalidation policy: lazy mode1389client # [ 0.489582] NetLabel: domain hash size = 1281390worker2 # [ 0.474513] ACPI: bus type USB registered1391worker1 # [ 0.464558] ACPI: PCI: Interrupt link LNKB configured for IRQ 101392client # [ 0.489936] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1393worker2 # [ 0.475249] usbcore: registered new interface driver usbfs1394lb # [ 0.461874] ACPI: PCI: Interrupt link LNKA configured for IRQ 101395worker1 # [ 0.465551] ACPI: PCI: Interrupt link LNKC configured for IRQ 111396client # [ 0.490905] NetLabel: unlabeled traffic allowed by default1397worker2 # [ 0.476139] usbcore: registered new interface driver hub1398lb # [ 0.462483] ACPI: PCI: Interrupt link LNKB configured for IRQ 101399worker1 # [ 0.466549] ACPI: PCI: Interrupt link LNKD configured for IRQ 111400client # [ 0.491613] PCI: Using ACPI for IRQ routing1401worker2 # [ 0.476903] usbcore: registered new device driver usb1402lb # [ 0.463476] ACPI: PCI: Interrupt link LNKC configured for IRQ 111403worker1 # [ 0.467550] ACPI: PCI: Interrupt link LNKE configured for IRQ 101404lb # [ 0.464470] ACPI: PCI: Interrupt link LNKD configured for IRQ 111405worker2 # [ 0.478153] NetLabel: Initializing1406worker1 # [ 0.468554] ACPI: PCI: Interrupt link LNKF configured for IRQ 101407worker2 # [ 0.478655] NetLabel: domain hash size = 1281408lb # [ 0.465469] ACPI: PCI: Interrupt link LNKE configured for IRQ 101409worker1 # [ 0.469585] ACPI: PCI: Interrupt link LNKG configured for IRQ 111410worker2 # [ 0.479241] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1411lb # [ 0.466480] ACPI: PCI: Interrupt link LNKF configured for IRQ 101412worker1 # [ 0.470555] ACPI: PCI: Interrupt link LNKH configured for IRQ 111413worker2 # [ 0.480205] NetLabel: unlabeled traffic allowed by default1414lb # [ 0.467478] ACPI: PCI: Interrupt link LNKG configured for IRQ 111415worker2 # [ 0.480969] PCI: Using ACPI for IRQ routing1416worker1 # [ 0.471488] ACPI: PCI: Interrupt link GSIA configured for IRQ 161417lb # [ 0.468472] ACPI: PCI: Interrupt link LNKH configured for IRQ 111418worker1 # [ 0.472467] ACPI: PCI: Interrupt link GSIB configured for IRQ 171419lb # [ 0.469423] ACPI: PCI: Interrupt link GSIA configured for IRQ 161420worker1 # [ 0.473470] ACPI: PCI: Interrupt link GSIC configured for IRQ 181421lb # [ 0.470393] ACPI: PCI: Interrupt link GSIB configured for IRQ 171422worker1 # [ 0.474467] ACPI: PCI: Interrupt link GSID configured for IRQ 191423lb # [ 0.471391] ACPI: PCI: Interrupt link GSIC configured for IRQ 181424worker1 # [ 0.475402] ACPI: PCI: Interrupt link GSIE configured for IRQ 201425lb # [ 0.472387] ACPI: PCI: Interrupt link GSID configured for IRQ 191426worker1 # [ 0.476180] ACPI: PCI: Interrupt link GSIF configured for IRQ 211427worker1 # [ 0.476468] ACPI: PCI: Interrupt link GSIG configured for IRQ 221428lb # [ 0.473359] ACPI: PCI: Interrupt link GSIE configured for IRQ 201429lb # [ 0.474139] ACPI: PCI: Interrupt link GSIF configured for IRQ 211430worker1 # [ 0.477405] ACPI: PCI: Interrupt link GSIH configured for IRQ 231431lb # [ 0.474390] ACPI: PCI: Interrupt link GSIG configured for IRQ 221432lb # [ 0.475388] ACPI: PCI: Interrupt link GSIH configured for IRQ 231433worker1 # [ 0.480017] iommu: Default domain type: Translated1434worker1 # [ 0.480469] iommu: DMA domain TLB invalidation policy: lazy mode1435worker1 # [ 0.481819] ACPI: bus type USB registered1436worker1 # [ 0.482553] usbcore: registered new interface driver usbfs1437lb # [ 0.477685] iommu: Default domain type: Translated1438worker1 # [ 0.483435] usbcore: registered new interface driver hub1439lb # [ 0.478389] iommu: DMA domain TLB invalidation policy: lazy mode1440worker1 # [ 0.484250] usbcore: registered new device driver usb1441lb # [ 0.479669] ACPI: bus type USB registered1442lb # [ 0.480509] usbcore: registered new interface driver usbfs1443worker1 # [ 0.486103] NetLabel: Initializing1444lb # [ 0.481393] usbcore: registered new interface driver hub1445worker1 # [ 0.486456] NetLabel: domain hash size = 1281446lb # [ 0.482282] usbcore: registered new device driver usb1447worker1 # [ 0.487217] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1448worker1 # [ 0.487509] NetLabel: unlabeled traffic allowed by default1449lb # [ 0.484045] NetLabel: Initializing1450worker1 # [ 0.488419] PCI: Using ACPI for IRQ routing1451lb # [ 0.484377] NetLabel: domain hash size = 1281452lb # [ 0.485132] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1453lb # [ 0.485436] NetLabel: unlabeled traffic allowed by default1454lb # [ 0.486387] PCI: Using ACPI for IRQ routing1455client # [ 0.535978] pci 0000:00:01.0: vgaarb: setting as boot VGA device1456client # [ 0.536932] pci 0000:00:01.0: vgaarb: bridge control possible1457client # [ 0.536932] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1458client # [ 0.536940] vgaarb: loaded1459client # [ 0.537691] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01460client # [ 0.537938] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1461client # [ 0.543026] clocksource: Switched to clocksource kvm-clock1462client # [ 0.544710] VFS: Disk quotas dquot_6.6.01463worker2 # [ 0.531924] pci 0000:00:01.0: vgaarb: setting as boot VGA device1464worker2 # [ 0.532237] pci 0000:00:01.0: vgaarb: bridge control possible1465client # [ 0.547267] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1466worker2 # [ 0.532237] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1467client # [ 0.548644] pnp: PnP ACPI init1468worker2 # [ 0.532255] vgaarb: loaded1469client # [ 0.549484] ACPI: IRQ 4 override to edge(!), high(!)1470worker2 # [ 0.533036] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01471worker2 # [ 0.533864] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1472client # [ 0.550445] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1473client # [ 0.552033] pnp: PnP ACPI: found 5 devices1474worker2 # [ 0.538341] clocksource: Switched to clocksource kvm-clock1475worker1 # [ 0.531872] pci 0000:00:01.0: vgaarb: setting as boot VGA device1476worker1 # [ 0.532451] pci 0000:00:01.0: vgaarb: bridge control possible1477worker2 # [ 0.540235] VFS: Disk quotas dquot_6.6.01478worker1 # [ 0.532451] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1479lb # [ 0.530577] pci 0000:00:01.0: vgaarb: setting as boot VGA device1480worker1 # [ 0.532460] vgaarb: loaded1481lb # [ 0.531372] pci 0000:00:01.0: vgaarb: bridge control possible1482client # [ 0.559677] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1483worker1 # [ 0.533261] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01484worker2 # [ 0.542860] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1485lb # [ 0.531372] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1486client # [ 0.561107] clocksource: Switched to clocksource acpi_pm1487worker1 # [ 0.534034] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1488worker2 # [ 0.544189] pnp: PnP ACPI init1489lb # [ 0.531380] vgaarb: loaded1490client # [ 0.562020] NET: Registered PF_INET protocol family1491worker2 # [ 0.545020] ACPI: IRQ 4 override to edge(!), high(!)1492lb # [ 0.532119] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01493lb # [ 0.532378] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1494client # [ 0.563335] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1495worker2 # [ 0.546005] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1496worker2 # [ 0.547553] pnp: PnP ACPI: found 5 devices1497worker1 # [ 0.537562] clocksource: Switched to clocksource kvm-clock1498lb # [ 0.536465] clocksource: Switched to clocksource kvm-clock1499worker1 # [ 0.541382] VFS: Disk quotas dquot_6.6.01500lb # [ 0.538272] VFS: Disk quotas dquot_6.6.01501worker1 # [ 0.542101] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1502worker1 # [ 0.543574] pnp: PnP ACPI init1503worker1 # [ 0.544438] ACPI: IRQ 4 override to edge(!), high(!)1504lb # [ 0.540863] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1505lb # [ 0.542243] pnp: PnP ACPI init1506worker2 # [ 0.555211] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1507worker1 # [ 0.545406] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1508worker2 # [ 0.556685] clocksource: Switched to clocksource acpi_pm1509lb # [ 0.543069] ACPI: IRQ 4 override to edge(!), high(!)1510worker1 # [ 0.546936] pnp: PnP ACPI: found 5 devices1511worker2 # [ 0.557739] NET: Registered PF_INET protocol family1512lb # [ 0.544047] system 00:04: [mem 0xb0000000-0xbfffffff window] has been reserved1513lb # [ 0.545629] pnp: PnP ACPI: found 5 devices1514worker2 # [ 0.558975] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1515client # [ 0.581606] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1516worker1 # [ 0.554707] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1517worker1 # [ 0.556224] clocksource: Switched to clocksource acpi_pm1518client # [ 0.583545] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1519worker1 # [ 0.557246] NET: Registered PF_INET protocol family1520lb # [ 0.553232] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1521client # [ 0.584931] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1522lb # [ 0.554699] clocksource: Switched to clocksource acpi_pm1523worker1 # [ 0.558414] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1524client # [ 0.586276] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1525lb # [ 0.555648] NET: Registered PF_INET protocol family1526client # [ 0.587500] TCP: Hash tables configured (established 16384 bind 16384)1527lb # [ 0.556787] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1528client # [ 0.588722] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1529client # [ 0.590075] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1530client # [ 0.591244] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1531client # [ 0.592550] NET: Registered PF_UNIX/PF_LOCAL protocol family1532client # [ 0.593597] NET: Registered PF_XDP protocol family1533worker2 # [ 0.575944] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1534client # [ 0.594564] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1535worker2 # [ 0.577542] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1536client # [ 0.595611] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1537worker2 # [ 0.578863] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1538client # [ 0.596678] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1539client # [ 0.597820] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1540worker2 # [ 0.580264] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1541worker2 # [ 0.581536] TCP: Hash tables configured (established 16384 bind 16384)1542client # [ 0.598913] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1543client # [ 0.600090] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1544worker2 # [ 0.582674] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1545worker2 # [ 0.583962] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1546client # [ 0.601965] ACPI: \_SB_.GSIA: Enabled at IRQ 161547worker2 # [ 0.585092] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1548worker2 # [ 0.586372] NET: Registered PF_UNIX/PF_LOCAL protocol family1549worker1 # [ 0.576193] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1550worker2 # [ 0.587348] NET: Registered PF_XDP protocol family1551client # [ 0.604856] ACPI: \_SB_.GSIB: Enabled at IRQ 171552lb # [ 0.573964] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1553worker2 # [ 0.588165] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1554worker1 # [ 0.577647] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1555client # [ 0.606836] ACPI: \_SB_.GSIC: Enabled at IRQ 181556worker2 # [ 0.589165] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1557worker1 # [ 0.578924] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1558lb # [ 0.575503] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1559worker2 # [ 0.590150] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1560lb # [ 0.576803] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1561worker1 # [ 0.580241] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1562client # [ 0.608780] ACPI: \_SB_.GSID: Enabled at IRQ 191563worker2 # [ 0.591242] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1564worker1 # [ 0.581460] TCP: Hash tables configured (established 16384 bind 16384)1565lb # [ 0.578132] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1566worker2 # [ 0.592310] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1567client # [ 0.610494] PCI: CLS 0 bytes, default 641568worker1 # [ 0.582587] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1569lb # [ 0.579433] TCP: Hash tables configured (established 16384 bind 16384)1570worker2 # [ 0.593408] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1571worker1 # [ 0.583822] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1572lb # [ 0.580512] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1573client # [ 0.611430] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1574worker2 # [ 0.595083] ACPI: \_SB_.GSIA: Enabled at IRQ 161575worker1 # [ 0.584881] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1576lb # [ 0.581739] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1577client # [ 0.613184] Trying to unpack rootfs image as initramfs...1578worker1 # [ 0.586092] NET: Registered PF_UNIX/PF_LOCAL protocol family1579lb # [ 0.582832] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1580worker1 # [ 0.587076] NET: Registered PF_XDP protocol family1581worker2 # [ 0.597823] ACPI: \_SB_.GSIB: Enabled at IRQ 171582lb # [ 0.584078] NET: Registered PF_UNIX/PF_LOCAL protocol family1583worker1 # [ 0.587942] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1584lb # [ 0.585025] NET: Registered PF_XDP protocol family1585worker1 # [ 0.588963] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1586worker2 # [ 0.599839] ACPI: \_SB_.GSIC: Enabled at IRQ 181587lb # [ 0.585909] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1588worker1 # [ 0.590006] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1589lb # [ 0.586906] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1590worker1 # [ 0.591129] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1591worker2 # [ 0.601839] ACPI: \_SB_.GSID: Enabled at IRQ 191592lb # [ 0.587948] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1593worker1 # [ 0.592232] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1594lb # [ 0.589044] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1595worker2 # [ 0.603622] PCI: CLS 0 bytes, default 641596worker1 # [ 0.593311] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1597lb # [ 0.590189] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1598worker2 # [ 0.604521] Trying to unpack rootfs image as initramfs...1599lb # [ 0.591327] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1600worker1 # [ 0.595129] ACPI: \_SB_.GSIA: Enabled at IRQ 161601lb # [ 0.593153] ACPI: \_SB_.GSIA: Enabled at IRQ 161602worker1 # [ 0.598605] ACPI: \_SB_.GSIB: Enabled at IRQ 171603lb # [ 0.595880] ACPI: \_SB_.GSIB: Enabled at IRQ 171604worker1 # [ 0.600617] ACPI: \_SB_.GSIC: Enabled at IRQ 181605worker2 # [ 0.610707] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1606lb # [ 0.597879] ACPI: \_SB_.GSIC: Enabled at IRQ 181607worker1 # [ 0.602571] ACPI: \_SB_.GSID: Enabled at IRQ 191608lb # [ 0.600069] ACPI: \_SB_.GSID: Enabled at IRQ 191609worker1 # [ 0.604289] PCI: CLS 0 bytes, default 641610lb # [ 0.601841] PCI: CLS 0 bytes, default 641611worker1 # [ 0.605198] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1612lb # [ 0.602707] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1613worker1 # [ 0.606873] Trying to unpack rootfs image as initramfs...1614lb # [ 0.604446] Trying to unpack rootfs image as initramfs...1615client # [ 0.658260] Initialise system trusted keyrings1616client # [ 0.660809] workingset: timestamp_bits=40 max_order=19 bucket_order=01617worker2 # [ 0.652588] Initialise system trusted keyrings1618worker2 # [ 0.655563] workingset: timestamp_bits=40 max_order=19 bucket_order=01619worker1 # [ 0.652482] Initialise system trusted keyrings1620lb # [ 0.649274] Initialise system trusted keyrings1621worker1 # [ 0.655085] workingset: timestamp_bits=40 max_order=19 bucket_order=01622lb # [ 0.652359] workingset: timestamp_bits=40 max_order=19 bucket_order=01623client # [ 0.683835] Key type asymmetric registered1624client # [ 0.684645] Asymmetric key parser 'x509' registered1625client # [ 0.685628] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1626client # [ 0.688886] io scheduler mq-deadline registered1627client # [ 0.689700] io scheduler kyber registered1628client # [ 0.693769] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1629worker2 # [ 0.677505] Key type asymmetric registered1630client # [ 0.695131] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1631worker2 # [ 0.678229] Asymmetric key parser 'x509' registered1632worker2 # [ 0.679143] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1633client # [ 0.701433] Linux agpgart interface v0.1031634worker2 # [ 0.684613] io scheduler mq-deadline registered1635client # [ 0.702281] ACPI: bus type drm_connector registered1636worker2 # [ 0.685377] io scheduler kyber registered1637lb # [ 0.673612] Key type asymmetric registered1638worker1 # [ 0.677216] Key type asymmetric registered1639client # [ 0.705335] usbcore: registered new interface driver usbserial_generic1640worker1 # [ 0.677935] Asymmetric key parser 'x509' registered1641worker2 # [ 0.689003] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1642client # [ 0.706484] usbserial: USB Serial support registered for generic1643worker1 # [ 0.678820] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1644lb # [ 0.676323] Asymmetric key parser 'x509' registered1645worker2 # [ 0.690283] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1646lb # [ 0.677211] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1647client # [ 0.709770] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1648client # [ 0.709770] Please enable it if your BIOS has the CPPC option.1649lb # [ 0.680411] io scheduler mq-deadline registered1650worker1 # [ 0.684187] io scheduler mq-deadline registered1651lb # [ 0.681201] io scheduler kyber registered1652client # [ 0.712152] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1653worker1 # [ 0.684943] io scheduler kyber registered1654worker2 # [ 0.695468] Linux agpgart interface v0.1031655worker2 # [ 0.696211] ACPI: bus type drm_connector registered1656lb # [ 0.683852] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1657client # [ 0.715935] drop_monitor: Initializing network drop monitor service1658worker2 # [ 0.698552] usbcore: registered new interface driver usbserial_generic1659worker1 # [ 0.688644] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1660lb # [ 0.685202] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1661client # [ 0.717091] NET: Registered PF_INET6 protocol family1662worker2 # [ 0.699621] usbserial: USB Serial support registered for generic1663worker1 # [ 0.689874] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1664client # [ 0.720287] Segment Routing with IPv61665worker2 # [ 0.702426] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1666worker2 # [ 0.702426] Please enable it if your BIOS has the CPPC option.1667worker1 # [ 0.693809] Linux agpgart interface v0.1031668lb # [ 0.691264] Linux agpgart interface v0.1031669worker1 # [ 0.694638] ACPI: bus type drm_connector registered1670worker2 # [ 0.704734] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1671client # [ 0.722780] In-situ OAM (IOAM) with IPv61672lb # [ 0.692047] ACPI: bus type drm_connector registered1673client # [ 0.723901] IPI shorthand broadcast: enabled1674worker1 # [ 0.697509] usbcore: registered new interface driver usbserial_generic1675lb # [ 0.694813] usbcore: registered new interface driver usbserial_generic1676worker1 # [ 0.698645] usbserial: USB Serial support registered for generic1677lb # [ 0.695932] usbserial: USB Serial support registered for generic1678worker2 # [ 0.710553] drop_monitor: Initializing network drop monitor service1679worker2 # [ 0.711775] NET: Registered PF_INET6 protocol family1680worker1 # [ 0.702047] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1681worker1 # [ 0.702047] Please enable it if your BIOS has the CPPC option.1682worker2 # [ 0.714887] Segment Routing with IPv61683lb # [ 0.700320] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1684worker1 # [ 0.704370] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1685lb # [ 0.700320] Please enable it if your BIOS has the CPPC option.1686client # [ 0.732499] sched_clock: Marking stable (596015694, 135728731)->(805650051, -73905626)1687lb # [ 0.702528] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1688worker2 # [ 0.717444] In-situ OAM (IOAM) with IPv61689worker1 # [ 0.707166] drop_monitor: Initializing network drop monitor service1690worker2 # [ 0.718509] IPI shorthand broadcast: enabled1691worker1 # [ 0.708359] NET: Registered PF_INET6 protocol family1692lb # [ 0.705467] drop_monitor: Initializing network drop monitor service1693lb # [ 0.706650] NET: Registered PF_INET6 protocol family1694client # [ 0.737977] registered taskstats version 11695client # [ 0.738958] Loading compiled-in X.509 certificates1696lb # [ 0.709736] Segment Routing with IPv61697worker1 # [ 0.713336] Segment Routing with IPv61698worker1 # [ 0.713966] In-situ OAM (IOAM) with IPv61699worker1 # [ 0.715111] IPI shorthand broadcast: enabled1700lb # [ 0.712346] In-situ OAM (IOAM) with IPv61701worker2 # [ 0.726553] sched_clock: Marking stable (592014788, 134384038)->(804945895, -78547069)1702lb # [ 0.713486] IPI shorthand broadcast: enabled1703worker2 # [ 0.730651] registered taskstats version 11704worker2 # [ 0.731622] Loading compiled-in X.509 certificates1705worker1 # [ 0.723542] sched_clock: Marking stable (590016854, 133011725)->(794038359, -71009780)1706lb # [ 0.721823] sched_clock: Marking stable (589015973, 132283219)->(790966895, -69667703)1707worker1 # [ 0.728142] registered taskstats version 11708worker1 # [ 0.729182] Loading compiled-in X.509 certificates1709client # [ 0.757751] Demotion targets for Node 0: null1710lb # [ 0.727486] registered taskstats version 11711client # [ 0.758702] Key type .fscrypt registered1712lb # [ 0.728463] Loading compiled-in X.509 certificates1713client # [ 0.760756] Key type fscrypt-provisioning registered1714client # [ 0.761712] ima: No TPM chip found, activating TPM-bypass!1715client # [ 0.762691] ima: Allocated hash algorithm: sha11716client # [ 0.766798] ima: No architecture policies found1717client # [ 0.767840] PM: Magic number: 10:287:7871718worker2 # [ 0.750417] Demotion targets for Node 0: null1719worker2 # [ 0.751279] Key type .fscrypt registered1720client # [ 0.769625] RAS: Correctable Errors collector initialized.1721worker2 # [ 0.754403] Key type fscrypt-provisioning registered1722worker2 # [ 0.755351] ima: No TPM chip found, activating TPM-bypass!1723worker2 # [ 0.756308] ima: Allocated hash algorithm: sha11724worker1 # [ 0.748041] Demotion targets for Node 0: null1725worker2 # [ 0.758434] ima: No architecture policies found1726lb # [ 0.744982] Demotion targets for Node 0: null1727worker1 # [ 0.748931] Key type .fscrypt registered1728worker2 # [ 0.759351] PM: Magic number: 10:287:7871729client # [ 0.778466] clk: Disabling unused clocks1730lb # [ 0.748343] Key type .fscrypt registered1731worker1 # [ 0.752032] Key type fscrypt-provisioning registered1732lb # [ 0.749055] Key type fscrypt-provisioning registered1733worker2 # [ 0.763033] RAS: Correctable Errors collector initialized.1734worker1 # [ 0.752992] ima: No TPM chip found, activating TPM-bypass!1735lb # [ 0.750023] ima: No TPM chip found, activating TPM-bypass!1736worker1 # [ 0.753936] ima: Allocated hash algorithm: sha11737client # [ 0.781780] PM: genpd: Disabling unused power domains1738worker1 # [ 0.754769] ima: No architecture policies found1739lb # [ 0.752310] ima: Allocated hash algorithm: sha11740lb # [ 0.753143] ima: No architecture policies found1741worker1 # [ 0.757038] PM: Magic number: 10:287:7871742worker1 # [ 0.758680] RAS: Correctable Errors collector initialized.1743worker2 # [ 0.770652] clk: Disabling unused clocks1744lb # [ 0.757319] PM: Magic number: 10:287:7871745worker2 # [ 0.771361] PM: genpd: Disabling unused power domains1746lb # [ 0.759009] RAS: Correctable Errors collector initialized.1747worker1 # [ 0.767696] clk: Disabling unused clocks1748worker1 # [ 0.771044] PM: genpd: Disabling unused power domains1749lb # [ 0.767972] clk: Disabling unused clocks1750lb # [ 0.770323] PM: genpd: Disabling unused power domains1751worker2 # [ 0.936435] Freeing initrd memory: 29060K1752client # [ 0.954587] Freeing initrd memory: 29056K1753worker2 # [ 0.939736] Freeing unused decrypted memory: 2028K1754client # [ 0.958217] Freeing unused decrypted memory: 2028K1755worker2 # [ 0.943233] Freeing unused kernel image (initmem) memory: 3652K1756client # [ 0.961292] Freeing unused kernel image (initmem) memory: 3652K1757client # [ 0.962398] Write protecting the kernel read-only data: 32768k1758worker2 # [ 0.944783] Write protecting the kernel read-only data: 32768k1759client # [ 0.964473] Freeing unused kernel image (text/rodata gap) memory: 1184K1760worker2 # [ 0.947371] Freeing unused kernel image (text/rodata gap) memory: 1184K1761client # [ 0.966062] Freeing unused kernel image (rodata/data gap) memory: 720K1762worker2 # [ 0.949364] Freeing unused kernel image (rodata/data gap) memory: 720K1763worker1 # [ 0.942585] Freeing initrd memory: 29056K1764worker1 # [ 0.945798] Freeing unused decrypted memory: 2028K1765worker1 # [ 0.948536] Freeing unused kernel image (initmem) memory: 3652K1766worker1 # [ 0.949660] Write protecting the kernel read-only data: 32768k1767worker1 # [ 0.951574] Freeing unused kernel image (text/rodata gap) memory: 1184K1768worker1 # [ 0.953149] Freeing unused kernel image (rodata/data gap) memory: 720K1769lb # [ 0.971437] Freeing initrd memory: 29064K1770lb # [ 0.975658] Freeing unused decrypted memory: 2028K1771lb # [ 0.979200] Freeing unused kernel image (initmem) memory: 3652K1772lb # [ 0.981388] Write protecting the kernel read-only data: 32768k1773lb # [ 0.983871] Freeing unused kernel image (text/rodata gap) memory: 1184K1774lb # [ 0.985462] Freeing unused kernel image (rodata/data gap) memory: 720K1775client # [ 1.017056] x86/mm: Checked W+X mappings: passed, no W+X pages found.1776client # [ 1.018181] Run /init as init process1777worker2 # [ 1.005997] x86/mm: Checked W+X mappings: passed, no W+X pages found.1778worker2 # [ 1.007065] Run /init as init process1779client # [ 1.029959] systemd[1]: Inserted module 'autofs4'1780worker1 # [ 1.003769] x86/mm: Checked W+X mappings: passed, no W+X pages found.1781worker1 # [ 1.004871] Run /init as init process1782worker2 # [ 1.019493] systemd[1]: Inserted module 'autofs4'1783worker1 # [ 1.015655] systemd[1]: Inserted module 'autofs4'1784client # [ 1.050074] fuse: init (API version 7.45)1785worker2 # [ 1.039885] fuse: init (API version 7.45)1786client # [ 1.057648] ACPI: \_SB_.GSIG: Enabled at IRQ 221787client # [ 1.060354] ACPI: \_SB_.GSIH: Enabled at IRQ 231788worker1 # [ 1.035426] fuse: init (API version 7.45)1789client # [ 1.063973] ACPI: \_SB_.GSIE: Enabled at IRQ 201790worker2 # [ 1.047516] ACPI: \_SB_.GSIG: Enabled at IRQ 221791client # [ 1.066203] ACPI: \_SB_.GSIF: Enabled at IRQ 211792worker2 # [ 1.050176] ACPI: \_SB_.GSIH: Enabled at IRQ 231793lb # [ 1.036214] x86/mm: Checked W+X mappings: passed, no W+X pages found.1794lb # [ 1.037346] Run /init as init process1795worker1 # [ 1.042732] ACPI: \_SB_.GSIG: Enabled at IRQ 221796worker2 # [ 1.053524] ACPI: \_SB_.GSIE: Enabled at IRQ 201797client # [ 1.072350] virtiofs virtio5: discovered new tag: nix-store1798worker1 # [ 1.045198] ACPI: \_SB_.GSIH: Enabled at IRQ 231799worker2 # [ 1.055853] ACPI: \_SB_.GSIF: Enabled at IRQ 211800client # [ 1.073998] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1801worker1 # [ 1.048223] ACPI: \_SB_.GSIE: Enabled at IRQ 201802worker1 # [ 1.050223] ACPI: \_SB_.GSIF: Enabled at IRQ 211803worker2 # [ 1.062186] virtiofs virtio5: discovered new tag: nix-store1804client # [ 1.081483] virtiofs virtio6: discovered new tag: shared1805worker2 # [ 1.064154] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1806lb # [ 1.051643] systemd[1]: Inserted module 'autofs4'1807client # [ 1.083011] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1808worker1 # [ 1.055565] virtiofs virtio5: discovered new tag: nix-store1809worker1 # [ 1.057128] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1810client # [ 1.086363] virtiofs virtio7: discovered new tag: xchg1811client # [ 1.087842] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1812worker2 # [ 1.072375] virtiofs virtio6: discovered new tag: shared1813worker2 # [ 1.074573] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1814worker1 # [ 1.066040] virtiofs virtio6: discovered new tag: shared1815worker1 # [ 1.067720] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1816worker2 # [ 1.078837] virtiofs virtio7: discovered new tag: xchg1817worker2 # [ 1.081058] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1818worker1 # [ 1.071608] virtiofs virtio7: discovered new tag: xchg1819worker1 # [ 1.073292] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1820lb # [ 1.074355] fuse: init (API version 7.45)1821client # [ 1.109657] systemd[1]: Successfully made /usr/ read-only.1822lb # [ 1.081976] ACPI: \_SB_.GSIG: Enabled at IRQ 221823lb # [ 1.084409] ACPI: \_SB_.GSIH: Enabled at IRQ 231824lb # [ 1.087456] ACPI: \_SB_.GSIE: Enabled at IRQ 201825worker2 # [ 1.102535] systemd[1]: Successfully made /usr/ read-only.1826lb # [ 1.089787] ACPI: \_SB_.GSIF: Enabled at IRQ 211827worker1 # [ 1.095846] systemd[1]: Successfully made /usr/ read-only.1828lb # [ 1.095085] virtiofs virtio5: discovered new tag: nix-store1829lb # [ 1.096722] virtiofs virtio5: virtio_fs_setup_dax: No cache capability1830lb # [ 1.103757] virtiofs virtio6: discovered new tag: shared1831lb # [ 1.105538] virtiofs virtio6: virtio_fs_setup_dax: No cache capability1832lb # [ 1.109574] virtiofs virtio7: discovered new tag: xchg1833lb # [ 1.111062] virtiofs virtio7: virtio_fs_setup_dax: No cache capability1834lb # [ 1.132276] systemd[1]: Successfully made /usr/ read-only.1835client # [ 1.445278] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1836client # [ 1.450433] systemd[1]: Detected virtualization kvm.1837client # [ 1.451327] systemd[1]: Detected architecture x86-64.1838client # [ 1.452237] systemd[1]: Running in initrd.1839client # [ 1.453323] systemd[1]: Initializing machine ID from random generator.1840client # [ 1.454588] systemd[1]: Hostname set to <client>.1841worker2 # [ 1.438061] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1842worker1 # [ 1.431485] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1843worker2 # [ 1.449085] systemd[1]: Detected virtualization kvm.1844worker1 # [ 1.440325] systemd[1]: Detected virtualization kvm.1845worker2 # [ 1.451564] systemd[1]: Detected architecture x86-64.1846worker1 # [ 1.442791] systemd[1]: Detected architecture x86-64.1847worker2 # [ 1.454005] systemd[1]: Running in initrd.1848worker1 # [ 1.445247] systemd[1]: Running in initrd.1849worker2 # [ 1.456759] systemd[1]: Initializing machine ID from random generator.1850worker1 # [ 1.447841] systemd[1]: Initializing machine ID from random generator.1851worker2 # [ 1.459719] systemd[1]: Hostname set to <worker2>.1852worker1 # [ 1.451364] systemd[1]: Hostname set to <worker1>.1853lb # [ 1.468949] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1854lb # [ 1.481273] systemd[1]: Detected virtualization kvm.1855lb # [ 1.483548] systemd[1]: Detected architecture x86-64.1856lb # [ 1.485790] systemd[1]: Running in initrd.1857lb # [ 1.488433] systemd[1]: Initializing machine ID from random generator.1858lb # [ 1.491500] systemd[1]: Hostname set to <lb>.1859client # [ 1.656989] systemd[1]: bpf-restrict-fs: LSM BPF program attached1860worker2 # [ 1.672108] systemd[1]: bpf-restrict-fs: LSM BPF program attached1861worker1 # [ 1.675077] systemd[1]: bpf-restrict-fs: LSM BPF program attached1862client # [ 1.709007] systemd[1]: Queued start job for default target Initrd Default Target.1863client # [ 1.713049] systemd[1]: Created slice Slice /system/modprobe.1864client # [ 1.714297] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1865client # [ 1.715759] systemd[1]: Expecting device /dev/disk/by-label/nixos...1866client # [ 1.716874] systemd[1]: Reached target Path Units.1867client # [ 1.717774] systemd[1]: Reached target Slice Units.1868client # [ 1.718637] systemd[1]: Reached target Swaps.1869client # [ 1.719464] systemd[1]: Reached target Timer Units.1870client # [ 1.720451] systemd[1]: Listening on D-Bus System Message Bus Socket.1871client # [ 1.721680] systemd[1]: Listening on Journal Socket (/dev/log).1872client # [ 1.722907] systemd[1]: Listening on Journal Sockets.1873client # [ 1.723911] systemd[1]: Listening on udev Control Socket.1874client # [ 1.725006] systemd[1]: Listening on udev Kernel Socket.1875client # [ 1.725988] systemd[1]: Reached target Socket Units.1876client # [ 1.727772] systemd[1]: Starting Create List of Static Device Nodes...1877worker2 # [ 1.712018] systemd[1]: Queued start job for default target Initrd Default Target.1878client # [ 1.731553] systemd[1]: Starting Load Kernel Module configfs...1879worker2 # [ 1.715816] systemd[1]: Created slice Slice /system/modprobe.1880worker2 # [ 1.717003] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1881worker2 # [ 1.718492] systemd[1]: Expecting device /dev/disk/by-label/nixos...1882worker2 # [ 1.719611] systemd[1]: Reached target Path Units.1883worker2 # [ 1.720477] systemd[1]: Reached target Slice Units.1884worker2 # [ 1.721328] systemd[1]: Reached target Swaps.1885worker2 # [ 1.722131] systemd[1]: Reached target Timer Units.1886worker2 # [ 1.723120] systemd[1]: Listening on D-Bus System Message Bus Socket.1887worker2 # [ 1.724340] systemd[1]: Listening on Journal Socket (/dev/log).1888worker2 # [ 1.725523] systemd[1]: Listening on Journal Sockets.1889worker2 # [ 1.726478] systemd[1]: Listening on udev Control Socket.1890worker2 # [ 1.727504] systemd[1]: Listening on udev Kernel Socket.1891worker2 # [ 1.728413] systemd[1]: Reached target Socket Units.1892worker2 # [ 1.730242] systemd[1]: Starting Create List of Static Device Nodes...1893lb # [ 1.717652] systemd[1]: bpf-restrict-fs: LSM BPF program attached1894client # [ 1.739388] systemd[1]: Starting Journal Service...1895worker1 # [ 1.723339] systemd[1]: Queued start job for default target Initrd Default Target.1896worker2 # [ 1.734481] systemd[1]: Starting Load Kernel Module configfs...1897worker1 # [ 1.727363] systemd[1]: Created slice Slice /system/modprobe.1898worker1 # [ 1.728712] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1899worker1 # [ 1.730294] systemd[1]: Expecting device /dev/disk/by-label/nixos...1900worker1 # [ 1.731553] systemd[1]: Reached target Path Units.1901worker1 # [ 1.732553] systemd[1]: Reached target Slice Units.1902worker1 # [ 1.733531] systemd[1]: Reached target Swaps.1903worker1 # [ 1.734396] systemd[1]: Reached target Timer Units.1904worker1 # [ 1.735462] systemd[1]: Listening on D-Bus System Message Bus Socket.1905worker1 # [ 1.736818] systemd[1]: Listening on Journal Socket (/dev/log).1906client # [ 1.765939] systemd[1]: Starting Load Kernel Modules...1907worker1 # [ 1.738149] systemd[1]: Listening on Journal Sockets.1908worker1 # [ 1.739265] systemd[1]: Listening on udev Control Socket.1909worker1 # [ 1.740354] systemd[1]: Listening on udev Kernel Socket.1910worker1 # [ 1.741421] systemd[1]: Reached target Socket Units.1911client # [ 1.769837] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1912worker1 # [ 1.743321] systemd[1]: Starting Create List of Static Device Nodes...1913worker2 # [ 1.744489] systemd[1]: Starting Journal Service...1914worker1 # [ 1.748095] systemd[1]: Starting Load Kernel Module configfs...1915client # [ 1.776812] systemd[1]: Starting Coldplug All udev Devices...1916client # [ 1.788273] systemd-journald[66]: Collecting audit messages is disabled.1917worker2 # [ 1.772620] systemd[1]: Starting Load Kernel Modules...1918client # [ 1.792113] systemd[1]: Finished Create List of Static Device Nodes.1919worker2 # [ 1.776514] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1920client # [ 1.796045] systemd[1]: modprobe@configfs.service: Deactivated successfully.1921worker1 # [ 1.758162] systemd[1]: Starting Journal Service...1922lb # [ 1.772495] systemd[1]: Queued start job for default target Initrd Default Target.1923worker2 # [ 1.786645] systemd[1]: Starting Coldplug All udev Devices...1924client # [ 1.805035] systemd[1]: Finished Load Kernel Module configfs.1925lb # [ 1.776702] systemd[1]: Created slice Slice /system/modprobe.1926lb # [ 1.777928] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1927client # [ 1.809180] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1928lb # [ 1.779378] systemd[1]: Expecting device /dev/disk/by-label/nixos...1929worker1 # [ 1.783216] systemd[1]: Starting Load Kernel Modules...1930lb # [ 1.780508] systemd[1]: Reached target Path Units.1931lb # [ 1.781414] systemd[1]: Reached target Slice Units.1932lb # [ 1.782326] systemd[1]: Reached target Swaps.1933lb # [ 1.783139] systemd[1]: Reached target Timer Units.1934worker2 # [ 1.797507] systemd-journald[66]: Collecting audit messages is disabled.1935lb # [ 1.784162] systemd[1]: Listening on D-Bus System Message Bus Socket.1936worker1 # [ 1.787121] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1937lb # [ 1.785424] systemd[1]: Listening on Journal Socket (/dev/log).1938worker2 # [ 1.800042] systemd[1]: Finished Create List of Static Device Nodes.1939lb # [ 1.786610] systemd[1]: Listening on Journal Sockets.1940lb # [ 1.787591] systemd[1]: Listening on udev Control Socket.1941lb # [ 1.788651] systemd[1]: Listening on udev Kernel Socket.1942client # [ 1.819510] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1943lb # [ 1.789600] systemd[1]: Reached target Socket Units.1944lb # [ 1.791663] systemd[1]: Starting Create List of Static Device Nodes...1945worker2 # [ 1.806144] systemd[1]: modprobe@configfs.service: Deactivated successfully.1946worker1 # [ 1.796264] systemd[1]: Starting Coldplug All udev Devices...1947lb # [ 1.797407] systemd[1]: Starting Load Kernel Module configfs...1948worker2 # [ 1.813862] systemd[1]: Finished Load Kernel Module configfs.1949worker1 # [ 1.805635] systemd-journald[66]: Collecting audit messages is disabled.1950client # [ 1.832869] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1951worker1 # [ 1.809411] systemd[1]: Finished Create List of Static Device Nodes.1952worker2 # [ 1.819884] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1953client # [ 1.840983] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1954worker1 # [ 1.816659] systemd[1]: modprobe@configfs.service: Deactivated successfully.1955lb # [ 1.806543] systemd[1]: Starting Journal Service...1956worker2 # [ 1.832652] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1957worker1 # [ 1.824442] systemd[1]: Finished Load Kernel Module configfs.1958worker2 # [ 1.840739] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1959worker1 # [ 1.830461] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1960lb # [ 1.831950] systemd[1]: Starting Load Kernel Modules...1961client # [ 1.863309] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1962lb # [ 1.836488] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1963worker2 # [ 1.851655] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1964worker1 # [ 1.842097] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1965client # [ 1.874094] systemd[1]: Starting Create Static Device Nodes in /dev...1966lb # [ 1.846545] systemd[1]: Starting Coldplug All udev Devices...1967worker1 # [ 1.851307] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1968client # [ 1.883264] systemd[1]: Finished Load Kernel Modules.1969worker1 # [ 1.863059] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1970lb # [ 1.861709] systemd-journald[66]: Collecting audit messages is disabled.1971lb # [ 1.863072] systemd[1]: Finished Create List of Static Device Nodes.1972client # [ 1.894851] systemd[1]: Starting Apply Kernel Variables...1973lb # [ 1.869391] systemd[1]: modprobe@configfs.service: Deactivated successfully.1974worker2 # [ 1.884580] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1975client # [ 1.904036] systemd[1]: Started Journal Service.1976client # [ 1.765792] systemd-modules-load[68]: Inserted module 'dm_mod'1977client # [ 1.770953] systemd-modules-load[68]: Inserted module 'virtio_balloon'1978lb # [ 1.875381] systemd[1]: Finished Load Kernel Module configfs.1979client # [ 1.771959] systemd-modules-load[68]: Inserted module 'virtio_gpu'1980worker2 # [ 1.893692] systemd[1]: Starting Create Static Device Nodes in /dev...1981worker1 # [ 1.884613] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1982lb # [ 1.881707] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1983worker2 # [ 1.903975] systemd[1]: Finished Load Kernel Modules.1984worker1 # [ 1.895243] systemd[1]: Starting Create Static Device Nodes in /dev...1985lb # [ 1.892500] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1986client # [ 1.790084] systemd[1]: Finished Create Static Device Nodes in /dev.1987client # [ 1.791072] systemd[1]: Reached target Preparation for Local File Systems.1988client # [ 1.791993] systemd[1]: Reached target Local File Systems.1989worker2 # [ 1.913114] systemd[1]: Starting Apply Kernel Variables...1990client # [ 1.796097] systemd[1]: Starting Create System Files and Directories...1991lb # [ 1.902134] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1992client # [ 1.801105] systemd[1]: Starting Rule-based Manager for Device Events and Files...1993worker1 # [ 1.909169] systemd[1]: Finished Load Kernel Modules.1994worker2 # [ 1.782856] systemd-modules-load[68]: Inserted module 'dm_mod'1995worker2 # [ 1.786969] systemd-modules-load[68]: Inserted module 'virtio_balloon'1996worker2 # [ 1.789292] systemd-modules-load[68]: Inserted module 'virtio_gpu'1997worker2 # [ 1.924828] systemd[1]: Started Journal Service.1998client # [ 1.807645] systemd[1]: Finished Apply Kernel Variables.1999lb # [ 1.913496] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev2000worker1 # [ 1.918229] systemd[1]: Starting Apply Kernel Variables...
loading 2,854 hidden lines…
4855worker2 # deleting unused links...4856worker2 # note: hard linking is currently saving 0.0 KiB4857worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.18 seconds)4858client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&24859client # don't know how to build these paths:4860client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4861worker2 # [ 44.931423] nix-grpc-daemon[784]: ts=2026-09-22T13:45:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:364804862worker2 # [ 44.937429] nix-grpc-daemon[784]: ts=2026-09-22T13:45:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:364804863worker2 # [ 44.959429] nix-grpc-daemon[784]: ts=2026-09-22T13:46:00Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14864client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 >&2, in 0.20 seconds)4865client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath4866client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 outPath, in 0.10 seconds)4867lb: must succeed: curl -sf localhost:9901/clusters4868lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4869lb: must succeed: curl -sf localhost:9901/clusters4870lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4871worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4872worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4873(finished: subtest: repeat is answered Cached by the scheduler without building, in 0.78 seconds)4874subtest: inputs already in the cache are not sent to the scheduler4875lb: must succeed: curl -sf localhost:9901/clusters4876lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)4877lb: must succeed: curl -sf localhost:9901/clusters4878lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)4879worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4880worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.04 seconds)4881client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b >&24882worker2 # [ 45.507730] nix-grpc-daemon[784]: ts=2026-09-22T13:46:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:364804883client # don't know how to build these paths:4884client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4885worker2 # [ 45.516074] nix-grpc-daemon[784]: ts=2026-09-22T13:46:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:364804886worker2 # [ 45.528228] nix-grpc-daemon[784]: ts=2026-09-22T13:46:00Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04887worker2 # [ 45.532591] nix-grpc-daemon[784]: ts=2026-09-22T13:46:00Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14888worker1 # [ 45.527177] nix-grpc-daemon[782]: ts=2026-09-22T13:45:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427504889worker1 # [ 45.540958] nix-grpc-daemon[782]: ts=2026-09-22T13:45:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427504890worker1 # [ 45.554211] nix-grpc-daemon[862]: time=2026-09-22T13:45:59.826Z level=INFO msg="All 2 paths already cached"4891client # copying 1 paths to 'lb:50051'...4892client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4893worker1 # [ 45.563073] nix-grpc-daemon[782]: ts=2026-09-22T13:45:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:427504894worker1 # [ 45.578653] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)4895lb # [ 45.583202] envoy[917]: conn peer=[2001:db8:1::1]:37346 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=749 tx=423 ms=18944896lb # [ 45.588360] envoy[917]: conn peer=[2001:db8:1::1]:37356 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=604897lb # [ 45.593554] envoy[917]: conn peer=[2001:db8:1::1]:37370 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=104898lb # [ 45.597280] envoy[917]: conn peer=[2001:db8:1::1]:37376 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=84899lb # [ 45.602252] envoy[917]: conn peer=[2001:db8:1::1]:37382 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9395 tx=4327 ms=49584900lb # [ 45.606377] envoy[917]: conn peer=[2001:db8:1::1]:36586 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=434901worker1 # [ 45.685307] nix-daemon[848]: accepted connection from pid 994, user nix-grpc-daemon (trusted)4902worker1 # [ 45.698942] nix-daemon[848]: reaped child process 997, status = succeeded4903worker1 # [ 45.778944] nix-daemon[848]: accepted connection from pid 999, user nix-grpc-daemon (trusted)4904worker1 # [ 45.792321] nix-daemon[848]: reaped child process 1002, status = succeeded4905lb # [ 45.796790] niks3-server[1008]: 2026/09/22 13:46:00 INFO Received uploads request method=POST path=/api/pending_closures4906worker1 # [ 45.804113] nix-grpc-daemon[862]: time=2026-09-22T13:46:00.076Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4907worker1 # [ 45.805645] nix-grpc-daemon[862]: time=2026-09-22T13:46:00.078Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4908lb # [ 45.813086] niks3-server[1008]: 2026/09/22 13:46:00 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4909lb # [ 45.818837] niks3-server[1008]: 2026/09/22 13:46:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4910lb # [ 45.821793] niks3-server[1008]: 2026/09/22 13:46:00 INFO Signed narinfos id=7 count=14911worker1 # [ 45.824673] nix-grpc-daemon[862]: time=2026-09-22T13:46:00.097Z level=INFO msg="Uploading 1 narinfos"4912lb # [ 45.827142] niks3-server[1008]: 2026/09/22 13:46:00 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4913lb # [ 45.833131] niks3-server[1008]: 2026/09/22 13:46:00 INFO Completed upload id=74914worker1 # [ 45.835927] nix-grpc-daemon[862]: time=2026-09-22T13:46:00.108Z level=INFO msg="Upload complete. (244ms)"4915lb # [ 45.834817] niks3-server[1008]: 2026/09/22 13:46:00 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4916worker1 # [ 45.838117] nix-grpc-daemon[782]: ts=2026-09-22T13:46:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=0 paths=1 nar_bytes_in=7524917lb # [ 45.837543] niks3-server[1008]: 2026/09/22 13:46:00 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4918worker1 # [ 45.842901] nix-grpc-daemon[782]: ts=2026-09-22T13:46:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427504919worker1 # [ 45.848488] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)4920worker1 # [ 45.850712] nix-daemon[848]: reaped child process 992, status = succeeded4921worker1 # [ 45.855806] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)4922client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4923client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4924client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4925client # farm-top> LOG-farm-top-t1b4926worker1 # [ 47.298572] nix-daemon[848]: reaped child process 1006, status = succeeded4927worker1 # [ 47.366045] nix-daemon[848]: accepted connection from pid 1022, user nix-grpc-daemon (trusted)4928worker1 # [ 47.378816] nix-daemon[848]: reaped child process 1025, status = succeeded4929lb # [ 47.381788] niks3-server[1008]: 2026/09/22 13:46:02 INFO Received uploads request method=POST path=/api/pending_closures4930worker1 # [ 47.393627] nix-grpc-daemon[862]: time=2026-09-22T13:46:01.666Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4931worker1 # [ 47.395100] nix-grpc-daemon[862]: time=2026-09-22T13:46:01.667Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4932lb # [ 47.421689] niks3-server[1008]: 2026/09/22 13:46:02 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4933lb # [ 47.433127] niks3-server[1008]: 2026/09/22 13:46:02 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4934lb # [ 47.438531] niks3-server[1008]: 2026/09/22 13:46:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4935worker1 # [ 47.449441] nix-grpc-daemon[862]: time=2026-09-22T13:46:01.721Z level=INFO msg="Uploading 1 narinfos"4936lb # [ 47.447119] niks3-server[1008]: 2026/09/22 13:46:02 INFO Signed narinfos id=8 count=14937lb # [ 47.455292] niks3-server[1008]: 2026/09/22 13:46:02 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4938lb # [ 47.463854] niks3-server[1008]: 2026/09/22 13:46:02 INFO Completed upload id=84939worker1 # [ 47.466577] nix-grpc-daemon[862]: time=2026-09-22T13:46:01.739Z level=INFO msg="Upload complete. (202ms)"4940lb # [ 47.467834] niks3-server[1008]: 2026/09/22 13:46:02 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4941lb # [ 47.470391] niks3-server[1008]: 2026/09/22 13:46:02 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4942worker1 # [ 47.472760] nix-grpc-daemon[782]: ts=2026-09-22T13:46:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=1 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14943worker1 # [ 47.489766] nix-daemon[848]: reaped child process 1004, status = succeeded4944client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.15 seconds)4945client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b outPath4946client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.12 seconds)4947lb: must succeed: curl -sf localhost:9901/clusters4948lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4949lb: must succeed: curl -sf localhost:9901/clusters4950lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)4951worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4952worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4953(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.49 seconds)4954subtest: outputs read back through any worker carry the cache signature4955client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig >&24956client # don't know how to build these paths:4957client # /nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv4958worker2 # [ 47.871429] nix-grpc-daemon[784]: ts=2026-09-22T13:46:02Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:364804959worker1 # [ 47.866408] nix-grpc-daemon[782]: ts=2026-09-22T13:46:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427504960worker2 # [ 47.897252] nix-grpc-daemon[784]: ts=2026-09-22T13:46:02Z level=debug event=want client=4 drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv cached=04961worker2 # [ 47.903653] nix-grpc-daemon[784]: ts=2026-09-22T13:46:02Z level=debug event=assigned drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv worker=192.168.1.4:50051 clients=14962worker2 # [ 47.908720] nix-grpc-daemon[784]: ts=2026-09-22T13:46:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364804963worker2 # [ 47.920573] nix-grpc-daemon[784]: ts=2026-09-22T13:46:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:364804964client # copying 1 paths to 'lb:50051'...4965client # copying path '/nix/store/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv' to 'lb:50051'...4966worker2 # [ 47.932173] nix-grpc-daemon[784]: ts=2026-09-22T13:46:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:364804967worker2 # [ 47.944313] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4968worker2 # [ 47.964895] nix-grpc-daemon[784]: ts=2026-09-22T13:46:03Z level=info event=niks3_push_started pid=9224969worker2 # [ 48.091220] nix-grpc-daemon[922]: time=2026-09-22T13:46:03.152Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""4970worker2 # [ 48.181157] nix-daemon[870]: accepted connection from pid 935, user nix-grpc-daemon (trusted)4971worker2 # [ 48.196070] nix-daemon[870]: reaped child process 938, status = succeeded4972worker2 # [ 48.279375] nix-daemon[870]: accepted connection from pid 940, user nix-grpc-daemon (trusted)4973worker2 # [ 48.293269] nix-daemon[870]: reaped child process 943, status = succeeded4974lb # [ 48.287425] niks3-server[1008]: 2026/09/22 13:46:03 INFO Received uploads request method=POST path=/api/pending_closures4975worker2 # [ 48.306992] nix-grpc-daemon[922]: time=2026-09-22T13:46:03.369Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4976worker2 # [ 48.309023] nix-grpc-daemon[922]: time=2026-09-22T13:46:03.371Z level=INFO msg="Uploading lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv (504B)"4977lb # [ 48.320631] niks3-server[1008]: 2026/09/22 13:46:03 INFO Registered completed upload object_key=nar/0bddczcnpndyzwmafx9yqfqrkrp1jg9rr5hsixz6rpy99kg84g1g.nar.zst4978lb # [ 48.328131] niks3-server[1008]: 2026/09/22 13:46:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4979lb # [ 48.333251] niks3-server[1008]: 2026/09/22 13:46:03 INFO Signed narinfos id=9 count=14980worker2 # [ 48.348177] nix-grpc-daemon[922]: time=2026-09-22T13:46:03.410Z level=INFO msg="Uploading 1 narinfos"4981lb # [ 48.347763] niks3-server[1008]: 2026/09/22 13:46:03 INFO Registered completed upload object_key=lqi1rvcqjj8q9l279yzgrgaghp394zs5.ls4982lb # [ 48.352600] niks3-server[1008]: 2026/09/22 13:46:03 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4983lb # [ 48.359723] niks3-server[1008]: 2026/09/22 13:46:03 INFO Completed upload id=94984worker2 # [ 48.373659] nix-grpc-daemon[922]: time=2026-09-22T13:46:03.436Z level=INFO msg="Upload complete. (279ms)"4985lb # [ 48.364697] niks3-server[1008]: 2026/09/22 13:46:03 INFO Registered completed upload object_key=lqi1rvcqjj8q9l279yzgrgaghp394zs5.narinfo4986worker2 # [ 48.376135] nix-grpc-daemon[784]: ts=2026-09-22T13:46:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 paths=1 nar_bytes_in=5044987worker2 # [ 48.383745] nix-grpc-daemon[784]: ts=2026-09-22T13:46:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364804988worker2 # [ 48.394891] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4989worker2 # [ 48.397219] nix-daemon[870]: reaped child process 920, status = succeeded4990worker2 # [ 48.402493] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4991client # building '/nix/store/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv' on lb:50051...4992client # farm-a> worker2: building lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv4993client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4994client # farm-a> LOG-farm-a-sig4995worker2 # [ 49.218359] nix-daemon[870]: reaped child process 947, status = succeeded4996worker2 # [ 49.292872] nix-daemon[870]: accepted connection from pid 963, user nix-grpc-daemon (trusted)4997worker2 # [ 49.307236] nix-daemon[870]: reaped child process 966, status = succeeded4998lb # [ 49.301842] niks3-server[1008]: 2026/09/22 13:46:04 INFO Received uploads request method=POST path=/api/pending_closures4999worker2 # [ 49.322046] nix-grpc-daemon[922]: time=2026-09-22T13:46:04.384Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5000worker2 # [ 49.323554] nix-grpc-daemon[922]: time=2026-09-22T13:46:04.386Z level=INFO msg="Uploading qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a (128B)"5001lb # [ 49.336162] niks3-server[1008]: 2026/09/22 13:46:04 INFO Registered completed upload object_key=log/lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv5002lb # [ 49.349374] niks3-server[1008]: 2026/09/22 13:46:04 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst5003lb # [ 49.363663] niks3-server[1008]: 2026/09/22 13:46:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign5004lb # [ 49.367950] niks3-server[1008]: 2026/09/22 13:46:04 INFO Signed narinfos id=10 count=15005worker2 # [ 49.382706] nix-grpc-daemon[922]: time=2026-09-22T13:46:04.444Z level=INFO msg="Uploading 1 narinfos"5006lb # [ 49.377083] niks3-server[1008]: 2026/09/22 13:46:04 INFO Registered completed upload object_key=qi07z82gylcaha6pl0vfp8s3j6f2gl91.ls5007lb # [ 49.384650] niks3-server[1008]: 2026/09/22 13:46:04 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete5008lb # [ 49.388506] niks3-server[1008]: 2026/09/22 13:46:04 INFO Completed upload id=105009worker2 # [ 49.401578] nix-grpc-daemon[922]: time=2026-09-22T13:46:04.464Z level=INFO msg="Upload complete. (192ms)"5010lb # [ 49.391036] niks3-server[1008]: 2026/09/22 13:46:04 INFO Registered completed upload object_key=qi07z82gylcaha6pl0vfp8s3j6f2gl91.narinfo5011worker2 # [ 49.404567] nix-grpc-daemon[784]: ts=2026-09-22T13:46:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=1 drv=lqi1rvcqjj8q9l279yzgrgaghp394zs5-farm-a.drv assign_id=5 outputs=15012worker2 # [ 49.411148] nix-grpc-daemon[784]: ts=2026-09-22T13:46:04Z level=debug event=want client=4 drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv cached=05013worker2 # [ 49.413159] nix-grpc-daemon[784]: ts=2026-09-22T13:46:04Z level=debug event=assigned drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv worker=192.168.1.4:50051 clients=15014worker2 # [ 49.418747] nix-daemon[870]: reaped child process 945, status = succeeded5015worker2 # [ 49.454932] nix-grpc-daemon[784]: ts=2026-09-22T13:46:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364805016worker2 # [ 49.463505] nix-grpc-daemon[784]: ts=2026-09-22T13:46:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:364805017worker2 # [ 49.471548] nix-grpc-daemon[922]: time=2026-09-22T13:46:04.534Z level=INFO msg="All 1 paths already cached"5018client # copying 1 paths to 'lb:50051'...5019worker2 # [ 49.477066] nix-grpc-daemon[784]: ts=2026-09-22T13:46:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:364805020client # copying path '/nix/store/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv' to 'lb:50051'...5021worker2 # [ 49.481988] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5022worker2 # [ 49.571264] nix-daemon[870]: accepted connection from pid 972, user nix-grpc-daemon (trusted)5023worker2 # [ 49.586462] nix-daemon[870]: reaped child process 975, status = succeeded5024worker2 # [ 49.668914] nix-daemon[870]: accepted connection from pid 977, user nix-grpc-daemon (trusted)5025worker2 # [ 49.683233] nix-daemon[870]: reaped child process 980, status = succeeded5026lb # [ 49.677089] niks3-server[1008]: 2026/09/22 13:46:04 INFO Received uploads request method=POST path=/api/pending_closures5027worker2 # [ 49.696394] nix-grpc-daemon[922]: time=2026-09-22T13:46:04.759Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5028worker2 # [ 49.698386] nix-grpc-daemon[922]: time=2026-09-22T13:46:04.760Z level=INFO msg="Uploading 08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv (624B)"5029lb # [ 49.706150] niks3-server[1008]: 2026/09/22 13:46:04 INFO Registered completed upload object_key=nar/1m1mvw8bmb91glp26qj699lxqv98q2iz8wm4ik3miz0xj6vi3mw6.nar.zst5030lb # [ 49.717321] niks3-server[1008]: 2026/09/22 13:46:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign5031lb # [ 49.724824] niks3-server[1008]: 2026/09/22 13:46:04 INFO Signed narinfos id=11 count=15032worker2 # [ 49.739593] nix-grpc-daemon[922]: time=2026-09-22T13:46:04.802Z level=INFO msg="Uploading 1 narinfos"5033lb # [ 49.732114] niks3-server[1008]: 2026/09/22 13:46:04 INFO Registered completed upload object_key=08wvq73c5i2kpl6g578glyb1rcqhv2iw.ls5034lb # [ 49.743704] niks3-server[1008]: 2026/09/22 13:46:04 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete5035lb # [ 49.750706] niks3-server[1008]: 2026/09/22 13:46:04 INFO Completed upload id=115036worker2 # [ 49.762720] nix-grpc-daemon[922]: time=2026-09-22T13:46:04.825Z level=INFO msg="Upload complete. (273ms)"5037lb # [ 49.753621] niks3-server[1008]: 2026/09/22 13:46:04 INFO Registered completed upload object_key=08wvq73c5i2kpl6g578glyb1rcqhv2iw.narinfo5038worker2 # [ 49.766904] nix-grpc-daemon[784]: ts=2026-09-22T13:46:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 paths=1 nar_bytes_in=6245039worker2 # [ 49.771765] nix-grpc-daemon[784]: ts=2026-09-22T13:46:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364805040worker2 # [ 49.782062] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5041worker2 # [ 49.787925] nix-daemon[870]: reaped child process 970, status = succeeded5042worker2 # [ 49.792123] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5043client # building '/nix/store/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv' on lb:50051...5044client # farm-b> worker2: building 08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv5045client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5046client # farm-b> LOG-farm-b-sig5047worker2 # [ 50.609931] nix-daemon[870]: reaped child process 984, status = succeeded5048worker2 # [ 50.683804] nix-daemon[870]: accepted connection from pid 1000, user nix-grpc-daemon (trusted)5049worker2 # [ 50.696412] nix-daemon[870]: reaped child process 1003, status = succeeded5050lb # [ 50.690048] niks3-server[1008]: 2026/09/22 13:46:05 INFO Received uploads request method=POST path=/api/pending_closures5051worker2 # [ 50.708123] nix-grpc-daemon[922]: time=2026-09-22T13:46:05.770Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5052worker2 # [ 50.709609] nix-grpc-daemon[922]: time=2026-09-22T13:46:05.772Z level=INFO msg="Uploading 2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b (176B)"5053lb # [ 50.713166] niks3-server[1008]: 2026/09/22 13:46:05 INFO Registered completed upload object_key=nar/0y3xlf15z3z540fgrc9sgnn2rl0aj3w6x9mjqd50hhqgmaxbiivd.nar.zst5054lb # [ 50.716431] niks3-server[1008]: 2026/09/22 13:46:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign5055lb # [ 50.720072] niks3-server[1008]: 2026/09/22 13:46:05 INFO Signed narinfos id=12 count=15056worker2 # [ 50.731914] nix-grpc-daemon[922]: time=2026-09-22T13:46:05.794Z level=INFO msg="Uploading 1 narinfos"5057lb # [ 50.721144] niks3-server[1008]: 2026/09/22 13:46:05 INFO Registered completed upload object_key=log/08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv5058lb # [ 50.723264] niks3-server[1008]: 2026/09/22 13:46:05 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.ls5059lb # [ 50.728972] niks3-server[1008]: 2026/09/22 13:46:05 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete5060lb # [ 50.732055] niks3-server[1008]: 2026/09/22 13:46:05 INFO Completed upload id=125061worker2 # [ 50.743817] nix-grpc-daemon[922]: time=2026-09-22T13:46:05.806Z level=INFO msg="Upload complete. (145ms)"5062lb # [ 50.735245] niks3-server[1008]: 2026/09/22 13:46:05 INFO Registered completed upload object_key=2f96mrg6ljwjqld1y3yaibmb07jkz3by.narinfo5063worker2 # [ 50.747357] nix-grpc-daemon[784]: ts=2026-09-22T13:46:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 drv=08wvq73c5i2kpl6g578glyb1rcqhv2iw-farm-b.drv assign_id=6 outputs=15064worker2 # [ 50.753629] nix-grpc-daemon[784]: ts=2026-09-22T13:46:05Z level=debug event=want client=4 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv cached=05065worker2 # [ 50.755590] nix-grpc-daemon[784]: ts=2026-09-22T13:46:05Z level=debug event=assigned drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv worker=192.168.1.4:50051 clients=15066worker2 # [ 50.760724] nix-daemon[870]: reaped child process 982, status = succeeded5067worker2 # [ 50.796972] nix-grpc-daemon[784]: ts=2026-09-22T13:46:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364805068worker2 # [ 50.808785] nix-grpc-daemon[784]: ts=2026-09-22T13:46:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:364805069worker2 # [ 50.819547] nix-grpc-daemon[922]: time=2026-09-22T13:46:05.882Z level=INFO msg="All 2 paths already cached"5070client # copying 1 paths to 'lb:50051'...5071client # copying path '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' to 'lb:50051'...5072worker2 # [ 50.827343] nix-grpc-daemon[784]: ts=2026-09-22T13:46:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:364805073worker2 # [ 50.836312] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5074worker2 # [ 50.927469] nix-daemon[870]: accepted connection from pid 1009, user nix-grpc-daemon (trusted)5075worker2 # [ 50.942396] nix-daemon[870]: reaped child process 1012, status = succeeded5076worker2 # [ 51.025277] nix-daemon[870]: accepted connection from pid 1014, user nix-grpc-daemon (trusted)5077worker2 # [ 51.039035] nix-daemon[870]: reaped child process 1017, status = succeeded5078lb # [ 51.034356] niks3-server[1008]: 2026/09/22 13:46:06 INFO Received uploads request method=POST path=/api/pending_closures5079worker2 # [ 51.055875] nix-grpc-daemon[922]: time=2026-09-22T13:46:06.118Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5080worker2 # [ 51.057340] nix-grpc-daemon[922]: time=2026-09-22T13:46:06.118Z level=INFO msg="Uploading cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv (752B)"5081lb # [ 51.064644] niks3-server[1008]: 2026/09/22 13:46:06 INFO Registered completed upload object_key=nar/0w21abkvlzg18nqkl2x4zxs7d0376kbld1n0fkr68vcpqpq1g5zg.nar.zst5082lb # [ 51.074212] niks3-server[1008]: 2026/09/22 13:46:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign5083lb # [ 51.079354] niks3-server[1008]: 2026/09/22 13:46:06 INFO Signed narinfos id=13 count=15084worker2 # [ 51.094096] nix-grpc-daemon[922]: time=2026-09-22T13:46:06.156Z level=INFO msg="Uploading 1 narinfos"5085lb # [ 51.087567] niks3-server[1008]: 2026/09/22 13:46:06 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.ls5086lb # [ 51.096813] niks3-server[1008]: 2026/09/22 13:46:06 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete5087lb # [ 51.101342] niks3-server[1008]: 2026/09/22 13:46:06 INFO Completed upload id=135088worker2 # [ 51.113143] nix-grpc-daemon[922]: time=2026-09-22T13:46:06.175Z level=INFO msg="Upload complete. (268ms)"5089worker2 # [ 51.115263] nix-grpc-daemon[784]: ts=2026-09-22T13:46:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 paths=1 nar_bytes_in=7525090lb # [ 51.104572] niks3-server[1008]: 2026/09/22 13:46:06 INFO Registered completed upload object_key=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx.narinfo5091worker2 # [ 51.118509] nix-grpc-daemon[784]: ts=2026-09-22T13:46:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364805092worker2 # [ 51.127409] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5093worker2 # [ 51.128734] nix-daemon[870]: reaped child process 1007, status = succeeded5094worker2 # [ 51.134070] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5095client # building '/nix/store/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv' on lb:50051...5096client # farm-top> worker2: building cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv5097client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5098client # farm-top> LOG-farm-top-sig5099worker2 # [ 52.179407] nix-daemon[870]: reaped child process 1021, status = succeeded5100worker2 # [ 52.248296] nix-daemon[870]: accepted connection from pid 1037, user nix-grpc-daemon (trusted)5101worker2 # [ 52.261664] nix-daemon[870]: reaped child process 1040, status = succeeded5102lb # [ 52.260605] niks3-server[1008]: 2026/09/22 13:46:07 INFO Received uploads request method=POST path=/api/pending_closures5103worker2 # [ 52.279745] nix-grpc-daemon[922]: time=2026-09-22T13:46:07.342Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5104worker2 # [ 52.281268] nix-grpc-daemon[922]: time=2026-09-22T13:46:07.343Z level=INFO msg="Uploading dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top (232B)"5105lb # [ 52.293549] niks3-server[1008]: 2026/09/22 13:46:07 INFO Registered completed upload object_key=log/cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv5106lb # [ 52.305555] niks3-server[1008]: 2026/09/22 13:46:07 INFO Registered completed upload object_key=nar/0h4zvhqjxsvdqli2l6i5ajh5sh40hz57gzchdirnvbisq2aks0k8.nar.zst5107lb # [ 52.317102] niks3-server[1008]: 2026/09/22 13:46:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign5108lb # [ 52.323544] niks3-server[1008]: 2026/09/22 13:46:07 INFO Signed narinfos id=14 count=15109worker2 # [ 52.335140] nix-grpc-daemon[922]: time=2026-09-22T13:46:07.397Z level=INFO msg="Uploading 1 narinfos"5110lb # [ 52.326326] niks3-server[1008]: 2026/09/22 13:46:07 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.ls5111lb # [ 52.333365] niks3-server[1008]: 2026/09/22 13:46:07 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete5112lb # [ 52.337238] niks3-server[1008]: 2026/09/22 13:46:07 INFO Completed upload id=145113worker2 # [ 52.349136] nix-grpc-daemon[922]: time=2026-09-22T13:46:07.411Z level=INFO msg="Upload complete. (207ms)"5114lb # [ 52.339341] niks3-server[1008]: 2026/09/22 13:46:07 INFO Registered completed upload object_key=dp6wdd8xfalk9qg1xqalcnk58hz7fnr5.narinfo5115worker2 # [ 52.355721] nix-grpc-daemon[784]: ts=2026-09-22T13:46:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=1 drv=cs7l7i2nvvh5pcyf2vglzb4mlwzjklnx-farm-top.drv assign_id=7 outputs=15116worker2 # [ 52.370985] nix-daemon[870]: reaped child process 1019, status = succeeded5117client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig >&2, in 4.65 seconds)5118client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig outPath5119client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag sig outPath, in 0.12 seconds)5120client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5121worker1 # [ 52.617871] nix-grpc-daemon[782]: ts=2026-09-22T13:46:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:536625122client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked):5123client # /nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a5124client # /nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b5125client # /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5126worker1 # [ 52.645588] nix-grpc-daemon[782]: ts=2026-09-22T13:46:06Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:536625127worker1 # [ 52.652087] nix-grpc-daemon[782]: copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'http://lb:5751'...5128worker1 # [ 52.665835] nix-grpc-daemon[782]: copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'http://lb:5751'...5129worker1 # [ 52.678531] nix-grpc-daemon[782]: copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'http://lb:5751'...5130client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.21 seconds)5131client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5132worker2 # [ 52.840652] nix-grpc-daemon[784]: ts=2026-09-22T13:46:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:587485133worker2 # [ 52.847624] nix-grpc-daemon[784]: ts=2026-09-22T13:46:07Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:587485134client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&ca-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ca.pem' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.18 seconds)5135client: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top5136client # finding garbage collector roots...5137client # removing stale temporary roots file "/nix/var/nix/temproots/903"5138client # removing stale temporary roots file "/nix/var/nix/temproots/859"5139client # removing stale temporary roots file "/nix/var/nix/temproots/867"5140client # removing stale temporary roots file "/nix/var/nix/temproots/888"5141client # removing stale temporary roots file "/nix/var/nix/temproots/880"5142client # removing stale temporary roots file "/nix/var/nix/temproots/847"5143client # removing stale temporary roots file "/nix/var/nix/temproots/826"5144client # removing stale temporary roots file "/nix/var/nix/temproots/810"5145client # deleting specified paths...5146client # deleting '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top'5147client # deleting unused links...5148client # note: hard linking is currently saving 0.0 KiB5149worker2 # [ 53.137597] nix-grpc-daemon[784]: ts=2026-09-22T13:46:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:364805150worker2 # [ 53.144350] nix-grpc-daemon[784]: ts=2026-09-22T13:46:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:364805151worker2 # [ 53.149913] nix-grpc-daemon[922]: time=2026-09-22T13:46:08.211Z level=INFO msg="All 1 paths already cached"5152worker1 # [ 53.143499] nix-grpc-daemon[782]: ts=2026-09-22T13:46:07Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:427505153worker2 # [ 53.159420] nix-grpc-daemon[784]: ts=2026-09-22T13:46:08Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:364805154client # copying 3 paths...5155worker2 # [ 53.168700] nix-grpc-daemon[784]: ts=2026-09-22T13:46:08Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:364805156client # copying path '/nix/store/qi07z82gylcaha6pl0vfp8s3j6f2gl91-farm-a' from 'grpc://lb:50051'...5157worker1 # [ 53.173061] nix-grpc-daemon[782]: ts=2026-09-22T13:46:07Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:427505158worker2 # [ 53.187071] nix-grpc-daemon[784]: ts=2026-09-22T13:46:08Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:364805159worker1 # [ 53.178845] nix-grpc-daemon[782]: ts=2026-09-22T13:46:07Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=0 paths=1 nar_bytes_out=2325160worker2 # [ 53.195069] nix-grpc-daemon[784]: ts=2026-09-22T13:46:08Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:364805161worker2 # [ 53.198778] nix-grpc-daemon[784]: ts=2026-09-22T13:46:08Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 paths=1 nar_bytes_out=1765162worker2 # [ 53.203258] nix-grpc-daemon[784]: ts=2026-09-22T13:46:08Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 paths=1 nar_bytes_out=1285163client # copying path '/nix/store/2f96mrg6ljwjqld1y3yaibmb07jkz3by-farm-b' from 'grpc://lb:50051'...5164client # copying path '/nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top' from 'grpc://lb:50051'...5165client: (finished: must succeed: nix-store --delete /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/dp6wdd8xfalk9qg1xqalcnk58hz7fnr5-farm-top, in 0.37 seconds)5166(finished: subtest: outputs read back through any worker carry the cache signature, in 5.52 seconds)5167subtest: two clients wanting the same drv build it once5168client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup5169client # [ 53.308674] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.5170client # Running as unit: dedup1.service; invocation ID: 210881a480c84b20bd511df25eff66c35171client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 0.03 seconds)5172client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup5173worker2 # [ 53.423936] nix-grpc-daemon[784]: ts=2026-09-22T13:46:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:364805174client # [ 53.445578] nix[956]: don't know how to build these paths:5175client # [ 53.446488] nix[956]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5176worker2 # [ 53.435195] nix-grpc-daemon[784]: ts=2026-09-22T13:46:08Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05177worker2 # [ 53.439777] nix-grpc-daemon[784]: ts=2026-09-22T13:46:08Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.3:50051 clients=15178worker1 # [ 53.432955] nix-grpc-daemon[782]: ts=2026-09-22T13:46:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427505179worker1 # [ 53.442671] nix-grpc-daemon[782]: ts=2026-09-22T13:46:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427505180client # [ 53.476079] nix[956]: copying 1 paths to 'lb:50051'...5181client # [ 53.476815] nix[956]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...5182worker1 # [ 53.451537] nix-grpc-daemon[782]: ts=2026-09-22T13:46:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:427505183worker1 # [ 53.462088] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5184worker1 # [ 53.566531] nix-daemon[848]: accepted connection from pid 1032, user nix-grpc-daemon (trusted)5185worker1 # [ 53.581897] nix-daemon[848]: reaped child process 1035, status = succeeded5186worker1 # [ 53.661208] nix-daemon[848]: accepted connection from pid 1037, user nix-grpc-daemon (trusted)5187worker1 # [ 53.675622] nix-daemon[848]: reaped child process 1040, status = succeeded5188lb # [ 53.679338] niks3-server[1008]: 2026/09/22 13:46:08 INFO Received uploads request method=POST path=/api/pending_closures5189worker1 # [ 53.688241] nix-grpc-daemon[862]: time=2026-09-22T13:46:07.960Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5190worker1 # [ 53.690047] nix-grpc-daemon[862]: time=2026-09-22T13:46:07.962Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"5191lb # [ 53.696463] niks3-server[1008]: 2026/09/22 13:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign5192lb # [ 53.699418] niks3-server[1008]: 2026/09/22 13:46:08 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst5193lb # [ 53.702281] niks3-server[1008]: 2026/09/22 13:46:08 INFO Signed narinfos id=15 count=15194worker1 # [ 53.706412] nix-grpc-daemon[862]: time=2026-09-22T13:46:07.979Z level=INFO msg="Uploading 1 narinfos"5195lb # [ 53.704341] niks3-server[1008]: 2026/09/22 13:46:08 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls5196lb # [ 53.709097] niks3-server[1008]: 2026/09/22 13:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete5197lb # [ 53.712980] niks3-server[1008]: 2026/09/22 13:46:08 INFO Completed upload id=155198worker1 # [ 53.715837] nix-grpc-daemon[862]: time=2026-09-22T13:46:07.988Z level=INFO msg="Upload complete. (242ms)"5199worker1 # [ 53.717981] nix-grpc-daemon[782]: ts=2026-09-22T13:46:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=0 paths=1 nar_bytes_in=4325200lb # [ 53.715860] niks3-server[1008]: 2026/09/22 13:46:08 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo5201worker1 # [ 53.721065] nix-grpc-daemon[782]: ts=2026-09-22T13:46:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427505202worker1 # [ 53.726131] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5203worker1 # [ 53.732823] nix-daemon[848]: reaped child process 1030, status = succeeded5204client # [ 53.760097] nix[956]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5205worker1 # [ 53.735722] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5206client # [ 54.347856] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup.5207client # Running as unit: dedup2.service; invocation ID: 5383cf0578d1460698d3a8ab251fa84e5208client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag dedup, in 1.04 seconds)5209worker2 # [ 54.469764] nix-grpc-daemon[784]: ts=2026-09-22T13:46:09Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:364805210client # [ 54.491797] nix[972]: don't know how to build these paths:5211client # [ 54.492791] nix[972]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5212worker2 # [ 54.482413] nix-grpc-daemon[784]: ts=2026-09-22T13:46:09Z level=debug event=want client=6 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=05213worker1 # [ 54.478200] nix-grpc-daemon[782]: ts=2026-09-22T13:46:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427505214client # [ 54.505821] nix[972]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...5215client: must succeed: sleep 25216lb # [ 55.582890] envoy[917]: conn peer=[2001:db8:1::1]:36590 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4182 tx=1894 ms=19895217lb # [ 55.586745] envoy[917]: conn peer=[2001:db8:1::1]:36600 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9494 tx=4378 ms=45055218lb # [ 55.590576] envoy[917]: conn peer=[2001:db8:1::1]:41188 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1051 tx=2889 ms=1205219lb # [ 55.594963] envoy[917]: conn peer=[2001:db8:1::1]:41216 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=330 ms=645220lb # [ 55.598569] envoy[917]: conn peer=[2001:db8:1::1]:41212 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=408 tx=376 ms=695221lb # [ 55.602169] envoy[917]: conn peer=[2001:db8:1::1]:41204 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=410 ms=755222client: (finished: must succeed: sleep 2, in 2.03 seconds)5223client: waiting for success: ! systemctl is-active dedup1 dedup25224worker1 # [ 57.781666] nix-daemon[848]: reaped child process 1044, status = succeeded5225worker1 # [ 57.852739] nix-daemon[848]: accepted connection from pid 1074, user nix-grpc-daemon (trusted)5226lb # [ 57.863663] niks3-server[1008]: 2026/09/22 13:46:12 INFO Received uploads request method=POST path=/api/pending_closures5227worker1 # [ 57.866970] nix-daemon[848]: reaped child process 1077, status = succeeded5228worker1 # [ 57.871857] nix-grpc-daemon[862]: time=2026-09-22T13:46:12.144Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5229worker1 # [ 57.876145] nix-grpc-daemon[862]: time=2026-09-22T13:46:12.146Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"5230lb # [ 57.887332] niks3-server[1008]: 2026/09/22 13:46:13 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv5231lb # [ 57.892182] niks3-server[1008]: 2026/09/22 13:46:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign5232lb # [ 57.895084] niks3-server[1008]: 2026/09/22 13:46:13 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst5233lb # [ 57.898049] niks3-server[1008]: 2026/09/22 13:46:13 INFO Signed narinfos id=16 count=15234worker1 # [ 57.902347] nix-grpc-daemon[862]: time=2026-09-22T13:46:12.174Z level=INFO msg="Uploading 1 narinfos"5235lb # [ 57.901250] niks3-server[1008]: 2026/09/22 13:46:13 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls5236lb # [ 57.908195] niks3-server[1008]: 2026/09/22 13:46:13 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete5237lb # [ 57.914503] niks3-server[1008]: 2026/09/22 13:46:13 INFO Completed upload id=165238worker1 # [ 57.917347] nix-grpc-daemon[862]: time=2026-09-22T13:46:12.190Z level=INFO msg="Upload complete. (171ms)"5239lb # [ 57.916628] niks3-server[1008]: 2026/09/22 13:46:13 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo5240worker1 # [ 57.920194] nix-grpc-daemon[782]: ts=2026-09-22T13:46:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=15241worker1 # [ 57.923091] nix-grpc-daemon[782]: ts=2026-09-22T13:46:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=8 outputs=15242worker1 # [ 57.929761] nix-daemon[848]: reaped child process 1042, status = succeeded5243client # [ 57.997982] systemd[1]: dedup1.service: Deactivated successfully.5244client # [ 58.002139] systemd[1]: dedup1.service: Consumed 116ms CPU time over 4.692s wall clock time, 39.5M memory peak, 7.1K incoming IP traffic, 8.2K outgoing IP traffic.5245client # [ 58.010564] systemd[1]: dedup2.service: Deactivated successfully.5246client # [ 58.012720] systemd[1]: dedup2.service: Consumed 129ms CPU time over 3.662s wall clock time, 35.4M memory peak, 6K incoming IP traffic, 5.7K outgoing IP traffic.5247client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.09 seconds)5248client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }5249client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)5250worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5251worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.02 seconds)5252worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true5253worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)5254worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5255worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)5256worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5257worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5258(finished: subtest: two clients wanting the same drv build it once, in 5.60 seconds)5259subtest: --timeout and --max-silent-time reach the builder5260worker2 # [ 58.991069] nix-grpc-daemon[784]: ts=2026-09-22T13:46:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:364805261worker2 # [ 59.001749] nix-grpc-daemon[784]: ts=2026-09-22T13:46:14Z level=debug event=want client=7 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv cached=05262worker2 # [ 59.006628] nix-grpc-daemon[784]: ts=2026-09-22T13:46:14Z level=debug event=assigned drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv worker=192.168.1.4:50051 clients=15263worker2 # [ 59.013355] nix-grpc-daemon[784]: ts=2026-09-22T13:46:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364805264worker2 # [ 59.020887] nix-grpc-daemon[784]: ts=2026-09-22T13:46:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:364805265worker2 # [ 59.031918] nix-grpc-daemon[784]: ts=2026-09-22T13:46:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:364805266worker2 # [ 59.043393] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5267worker2 # [ 59.139865] nix-daemon[870]: accepted connection from pid 1072, user nix-grpc-daemon (trusted)5268worker2 # [ 59.155044] nix-daemon[870]: reaped child process 1075, status = succeeded5269worker2 # [ 59.235216] nix-daemon[870]: accepted connection from pid 1077, user nix-grpc-daemon (trusted)5270worker2 # [ 59.249084] nix-daemon[870]: reaped child process 1080, status = succeeded5271lb # [ 59.244612] niks3-server[1008]: 2026/09/22 13:46:14 INFO Received uploads request method=POST path=/api/pending_closures5272worker2 # [ 59.264756] nix-grpc-daemon[922]: time=2026-09-22T13:46:14.327Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5273worker2 # [ 59.266342] nix-grpc-daemon[922]: time=2026-09-22T13:46:14.328Z level=INFO msg="Uploading qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv (424B)"5274lb # [ 59.278518] niks3-server[1008]: 2026/09/22 13:46:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5275lb # [ 59.288764] niks3-server[1008]: 2026/09/22 13:46:14 INFO Signed narinfos id=17 count=15276worker2 # [ 59.303874] nix-grpc-daemon[922]: time=2026-09-22T13:46:14.366Z level=INFO msg="Uploading 1 narinfos"5277lb # [ 59.295303] niks3-server[1008]: 2026/09/22 13:46:14 INFO Registered completed upload object_key=nar/1s4sciw269ksj1s7ymwz5cy4dw9n7bxv3k8wqw69h694spdh4ha2.nar.zst5278lb # [ 59.302138] niks3-server[1008]: 2026/09/22 13:46:14 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.ls5279lb # [ 59.311790] niks3-server[1008]: 2026/09/22 13:46:14 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5280lb # [ 59.320387] niks3-server[1008]: 2026/09/22 13:46:14 INFO Completed upload id=175281worker2 # [ 59.332320] nix-grpc-daemon[922]: time=2026-09-22T13:46:14.394Z level=INFO msg="Upload complete. (278ms)"5282lb # [ 59.326352] niks3-server[1008]: 2026/09/22 13:46:14 INFO Registered completed upload object_key=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k.narinfo5283worker2 # [ 59.336133] nix-grpc-daemon[784]: ts=2026-09-22T13:46:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 paths=1 nar_bytes_in=4245284worker2 # [ 59.341132] nix-grpc-daemon[784]: ts=2026-09-22T13:46:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364805285worker2 # [ 59.352120] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5286worker2 # [ 59.357461] nix-daemon[870]: reaped child process 1070, status = succeeded5287worker2 # [ 59.362161] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5288worker2 # [ 65.177104] nix-grpc-daemon[784]: ts=2026-09-22T13:46:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=5 drv=qkzy7qnn48bqc3ll9lb2m9zsrx1dr87k-slow-tmot.drv assign_id=9 outputs=05289worker2 # [ 65.202727] nix-daemon[870]: reaped child process 1082, status = succeeded5290worker2 # [ 65.205241] nix-daemon[870]: reaped child process 1084, status = succeeded5291worker2 # [ 65.351608] nix-grpc-daemon[784]: ts=2026-09-22T13:46:20Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:364805292worker2 # [ 65.362177] nix-grpc-daemon[784]: ts=2026-09-22T13:46:20Z level=debug event=want client=8 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv cached=05293worker1 # [ 65.356459] nix-grpc-daemon[782]: ts=2026-09-22T13:46:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427505294worker2 # [ 65.366492] nix-grpc-daemon[784]: ts=2026-09-22T13:46:20Z level=debug event=assigned drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv worker=192.168.1.3:50051 clients=15295worker1 # [ 65.364593] nix-grpc-daemon[782]: ts=2026-09-22T13:46:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427505296worker1 # [ 65.371410] nix-grpc-daemon[782]: ts=2026-09-22T13:46:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:427505297worker1 # [ 65.377256] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5298worker1 # [ 65.463097] nix-daemon[848]: accepted connection from pid 1090, user nix-grpc-daemon (trusted)5299worker1 # [ 65.476806] nix-daemon[848]: reaped child process 1093, status = succeeded5300worker1 # [ 65.555618] nix-daemon[848]: accepted connection from pid 1095, user nix-grpc-daemon (trusted)5301worker1 # [ 65.569258] nix-daemon[848]: reaped child process 1098, status = succeeded5302lb # [ 65.571321] niks3-server[1008]: 2026/09/22 13:46:20 INFO Received uploads request method=POST path=/api/pending_closures5303worker1 # [ 65.582482] nix-grpc-daemon[862]: time=2026-09-22T13:46:19.854Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5304worker1 # [ 65.583867] nix-grpc-daemon[862]: time=2026-09-22T13:46:19.854Z level=INFO msg="Uploading fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv (424B)"5305lb # [ 65.583821] envoy[917]: conn peer=[2001:db8:1::1]:41240 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1440 tx=1197 ms=34855306lb # [ 65.588503] envoy[917]: conn peer=[2001:db8:1::1]:41226 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2981 tx=1519 ms=45235307lb # [ 65.593184] envoy[917]: conn peer=[2001:db8:1::1]:41248 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=960 ms=62365308lb # [ 65.610405] niks3-server[1008]: 2026/09/22 13:46:20 INFO Registered completed upload object_key=nar/013y5sy2r6v0qdwv0dzpgvdfy08bds4mfh2qpvbgjazzrg2v71hp.nar.zst5309lb # [ 65.628111] niks3-server[1008]: 2026/09/22 13:46:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5310lb # [ 65.631374] niks3-server[1008]: 2026/09/22 13:46:20 INFO Signed narinfos id=18 count=15311worker1 # [ 65.634610] nix-grpc-daemon[862]: time=2026-09-22T13:46:19.907Z level=INFO msg="Uploading 1 narinfos"5312lb # [ 65.637592] niks3-server[1008]: 2026/09/22 13:46:20 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.ls5313lb # [ 65.642048] niks3-server[1008]: 2026/09/22 13:46:20 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5314lb # [ 65.646547] niks3-server[1008]: 2026/09/22 13:46:20 INFO Completed upload id=185315worker1 # [ 65.649343] nix-grpc-daemon[862]: time=2026-09-22T13:46:19.922Z level=INFO msg="Upload complete. (267ms)"5316lb # [ 65.648701] niks3-server[1008]: 2026/09/22 13:46:20 INFO Registered completed upload object_key=fvgcnys8vngcgklsp349d5pjxr4c66sg.narinfo5317worker1 # [ 65.651948] nix-grpc-daemon[782]: ts=2026-09-22T13:46:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=0 paths=1 nar_bytes_in=4245318worker1 # [ 65.655636] nix-grpc-daemon[782]: ts=2026-09-22T13:46:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427505319worker1 # [ 65.661295] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5320worker1 # [ 65.668051] nix-daemon[848]: reaped child process 1088, status = succeeded5321worker1 # [ 65.671084] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5322worker1 # [ 71.283684] nix-grpc-daemon[782]: ts=2026-09-22T13:46:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=5 drv=fvgcnys8vngcgklsp349d5pjxr4c66sg-slow-tmom.drv assign_id=10 outputs=05323worker1 # [ 71.309618] nix-daemon[848]: reaped child process 1100, status = succeeded5324worker1 # [ 71.313335] nix-daemon[848]: reaped child process 1102, status = succeeded5325(finished: subtest: --timeout and --max-silent-time reach the builder, in 12.56 seconds)5326subtest: the first client leaving does not fail the second5327client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share5328client # [ 71.491637] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share.5329client # Running as unit: share1.service; invocation ID: 0c2d928566c04d79873d03d3b5e77f515330client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share, in 0.06 seconds)5331client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share5332client # [ 71.635793] nix[1046]: don't know how to build these paths:5333client # [ 71.636763] nix[1046]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5334worker1 # [ 71.608994] nix-grpc-daemon[782]: ts=2026-09-22T13:46:25Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:427505335worker2 # [ 71.627931] nix-grpc-daemon[784]: ts=2026-09-22T13:46:26Z level=debug event=want client=9 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05336worker2 # [ 71.633784] nix-grpc-daemon[784]: ts=2026-09-22T13:46:26Z level=debug event=assigned drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv worker=192.168.1.4:50051 clients=15337worker2 # [ 71.640474] nix-grpc-daemon[784]: ts=2026-09-22T13:46:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364805338worker2 # [ 71.651244] nix-grpc-daemon[784]: ts=2026-09-22T13:46:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:364805339client # [ 71.676086] nix[1046]: copying 1 paths to 'lb:50051'...5340worker2 # [ 71.660385] nix-grpc-daemon[784]: ts=2026-09-22T13:46:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:364805341client # [ 71.676871] nix[1046]: copying path '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' to 'lb:50051'...5342worker2 # [ 71.666724] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5343worker2 # [ 71.751602] nix-daemon[870]: accepted connection from pid 1111, user nix-grpc-daemon (trusted)5344worker2 # [ 71.766946] nix-daemon[870]: reaped child process 1114, status = succeeded5345worker2 # [ 71.852207] nix-daemon[870]: accepted connection from pid 1116, user nix-grpc-daemon (trusted)5346worker2 # [ 71.866320] nix-daemon[870]: reaped child process 1119, status = succeeded5347lb # [ 71.862884] niks3-server[1008]: 2026/09/22 13:46:26 INFO Received uploads request method=POST path=/api/pending_closures5348worker2 # [ 71.882974] nix-grpc-daemon[922]: time=2026-09-22T13:46:26.945Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5349worker2 # [ 71.884445] nix-grpc-daemon[922]: time=2026-09-22T13:46:26.945Z level=INFO msg="Uploading j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv (432B)"5350lb # [ 71.894176] niks3-server[1008]: 2026/09/22 13:46:27 INFO Registered completed upload object_key=nar/0s0b0gfhn36p0aylk1qa64cgw01bi7a8ca93iv4zg2q7wqcjcqhs.nar.zst5351lb # [ 71.908483] niks3-server[1008]: 2026/09/22 13:46:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5352lb # [ 71.914833] niks3-server[1008]: 2026/09/22 13:46:27 INFO Signed narinfos id=19 count=15353worker2 # [ 71.930334] nix-grpc-daemon[922]: time=2026-09-22T13:46:26.992Z level=INFO msg="Uploading 1 narinfos"5354lb # [ 71.929491] niks3-server[1008]: 2026/09/22 13:46:27 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.ls5355lb # [ 71.936967] niks3-server[1008]: 2026/09/22 13:46:27 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5356lb # [ 71.944391] niks3-server[1008]: 2026/09/22 13:46:27 INFO Completed upload id=195357lb # [ 71.946296] niks3-server[1008]: 2026/09/22 13:46:27 INFO Registered completed upload object_key=j2xs6vfpc2dkbabf87x31z8q4j8chjfw.narinfo5358worker2 # [ 71.957758] nix-grpc-daemon[922]: time=2026-09-22T13:46:27.020Z level=INFO msg="Upload complete. (286ms)"5359worker2 # [ 71.962374] nix-grpc-daemon[784]: ts=2026-09-22T13:46:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 paths=1 nar_bytes_in=4325360worker2 # [ 71.967279] nix-grpc-daemon[784]: ts=2026-09-22T13:46:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364805361worker2 # [ 71.977529] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5362worker2 # [ 71.988124] nix-daemon[870]: reaped child process 1109, status = succeeded5363worker2 # [ 71.991102] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5364client # [ 72.007145] nix[1046]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5365client # [ 72.556928] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share.5366client # Running as unit: share2.service; invocation ID: 5eac953f781740bbace1b934ece578295367client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag share, in 1.07 seconds)5368worker2 # [ 72.684518] nix-grpc-daemon[784]: ts=2026-09-22T13:46:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:364805369client # [ 72.708064] nix[1062]: this derivation will be built:5370worker2 # [ 72.692632] nix-grpc-daemon[784]: ts=2026-09-22T13:46:27Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:364805371client # [ 72.711815] nix[1062]: /nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5372worker2 # [ 72.700433] nix-grpc-daemon[784]: ts=2026-09-22T13:46:27Z level=debug event=want client=10 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv cached=05373worker2 # [ 72.702611] nix-grpc-daemon[784]: ts=2026-09-22T13:46:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364805374client # [ 72.720884] nix[1062]: building '/nix/store/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv' on lb:50051...5375client: must succeed: sleep 2; systemctl kill -s INT share15376lb # [ 75.584833] envoy[917]: conn peer=[2001:db8:1::1]:34022 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2978 tx=933 ms=59945377client # [ 75.807237] systemd[1]: share1.service: Sent signal SIGINT to main process 1046 (nix) on client request.5378client # [ 75.809926] nix[1046]: shutting downj2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5379client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.04 seconds)5380client: waiting for success: ! systemctl is-active share15381client # [ 75.814056] nix[1046]: error: interrupted by the user5382client # [ 75.838567] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE5383client # [ 75.840215] systemd[1]: share1.service: Failed with result 'exit-code'.5384client # [ 75.842059] systemd[1]: share1.service: Consumed 114ms CPU time over 4.349s wall clock time, 41.5M memory peak, 6.2K incoming IP traffic, 8.1K outgoing IP traffic.5385client: (finished: waiting for success: ! systemctl is-active share1, in 1.06 seconds)5386client: waiting for success: ! systemctl is-active share25387worker2 # [ 77.235395] nix-daemon[870]: reaped child process 1123, status = succeeded5388worker2 # [ 77.315334] nix-daemon[870]: accepted connection from pid 1159, user nix-grpc-daemon (trusted)5389worker2 # [ 77.329515] nix-daemon[870]: reaped child process 1162, status = succeeded5390lb # [ 77.325408] niks3-server[1008]: 2026/09/22 13:46:32 INFO Received uploads request method=POST path=/api/pending_closures5391worker2 # [ 77.347652] nix-grpc-daemon[922]: time=2026-09-22T13:46:32.410Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5392worker2 # [ 77.349209] nix-grpc-daemon[922]: time=2026-09-22T13:46:32.411Z level=INFO msg="Uploading b9lg6znd0rinl36gy6ixq1dcj6b8bhhw-slow-share (120B)"5393lb # [ 77.365448] niks3-server[1008]: 2026/09/22 13:46:32 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst5394lb # [ 77.380147] niks3-server[1008]: 2026/09/22 13:46:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5395lb # [ 77.384648] niks3-server[1008]: 2026/09/22 13:46:32 INFO Signed narinfos id=20 count=15396worker2 # [ 77.399249] nix-grpc-daemon[922]: time=2026-09-22T13:46:32.461Z level=INFO msg="Uploading 1 narinfos"5397lb # [ 77.391596] niks3-server[1008]: 2026/09/22 13:46:32 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.ls5398lb # [ 77.397437] niks3-server[1008]: 2026/09/22 13:46:32 INFO Registered completed upload object_key=log/j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv5399lb # [ 77.403252] niks3-server[1008]: 2026/09/22 13:46:32 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5400lb # [ 77.405789] niks3-server[1008]: 2026/09/22 13:46:32 INFO Completed upload id=205401lb # [ 77.408780] niks3-server[1008]: 2026/09/22 13:46:32 INFO Registered completed upload object_key=b9lg6znd0rinl36gy6ixq1dcj6b8bhhw.narinfo5402worker2 # [ 77.418910] nix-grpc-daemon[922]: time=2026-09-22T13:46:32.481Z level=INFO msg="Upload complete. (214ms)"5403worker2 # [ 77.427320] nix-grpc-daemon[784]: ts=2026-09-22T13:46:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=4 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15404worker2 # [ 77.433267] nix-grpc-daemon[784]: ts=2026-09-22T13:46:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=5 drv=j2xs6vfpc2dkbabf87x31z8q4j8chjfw-slow-share.drv assign_id=11 outputs=15405worker2 # [ 77.445525] nix-daemon[870]: reaped child process 1121, status = succeeded5406client # [ 77.513313] systemd[1]: share2.service: Deactivated successfully.5407client # [ 77.516980] systemd[1]: share2.service: Consumed 123ms CPU time over 4.959s wall clock time, 35.4M memory peak, 6.6K incoming IP traffic, 6K outgoing IP traffic.5408client: (finished: waiting for success: ! systemctl is-active share2, in 1.08 seconds)5409client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }5410client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.03 seconds)5411(finished: subtest: the first client leaving does not fail the second, in 6.72 seconds)5412subtest: build hook: nix-daemon with builders = grpc://lb5413client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log5414client # [ 78.260458] systemd[1]: Started Nix Daemon.5415client # [ 78.350275] nix-daemon[1114]: accepted connection from pid 1111, user root (trusted)5416worker2 # [ 78.452583] nix-grpc-daemon[784]: ts=2026-09-22T13:46:33Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:364805417worker1 # [ 78.448102] nix-grpc-daemon[782]: ts=2026-09-22T13:46:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427505418worker2 # [ 78.464879] nix-grpc-daemon[784]: ts=2026-09-22T13:46:33Z level=debug event=want client=11 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=05419worker2 # [ 78.467025] nix-grpc-daemon[784]: ts=2026-09-22T13:46:33Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=15420worker1 # [ 78.462928] nix-grpc-daemon[782]: ts=2026-09-22T13:46:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427505421worker1 # [ 78.474887] nix-grpc-daemon[782]: ts=2026-09-22T13:46:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427505422worker1 # [ 78.486130] nix-grpc-daemon[782]: ts=2026-09-22T13:46:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:427505423worker1 # [ 78.496899] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5424worker1 # [ 78.592475] nix-daemon[848]: accepted connection from pid 1141, user nix-grpc-daemon (trusted)5425worker1 # [ 78.605102] nix-daemon[848]: reaped child process 1144, status = succeeded5426worker1 # [ 78.679085] nix-daemon[848]: accepted connection from pid 1146, user nix-grpc-daemon (trusted)5427worker1 # [ 78.692807] nix-daemon[848]: reaped child process 1149, status = succeeded5428lb # [ 78.690580] niks3-server[1008]: 2026/09/22 13:46:33 INFO Received uploads request method=POST path=/api/pending_closures5429worker1 # [ 78.702183] nix-grpc-daemon[862]: time=2026-09-22T13:46:32.974Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5430worker1 # [ 78.703990] nix-grpc-daemon[862]: time=2026-09-22T13:46:32.976Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"5431lb # [ 78.715981] niks3-server[1008]: 2026/09/22 13:46:33 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst5432lb # [ 78.732734] niks3-server[1008]: 2026/09/22 13:46:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5433worker1 # [ 78.742170] nix-grpc-daemon[862]: time=2026-09-22T13:46:33.014Z level=INFO msg="Uploading 1 narinfos"5434lb # [ 78.739193] niks3-server[1008]: 2026/09/22 13:46:33 INFO Signed narinfos id=21 count=15435lb # [ 78.745993] niks3-server[1008]: 2026/09/22 13:46:33 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls5436lb # [ 78.755784] niks3-server[1008]: 2026/09/22 13:46:33 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5437lb # [ 78.761052] niks3-server[1008]: 2026/09/22 13:46:33 INFO Completed upload id=215438worker1 # [ 78.763434] nix-grpc-daemon[862]: time=2026-09-22T13:46:33.036Z level=INFO msg="Upload complete. (255ms)"5439worker1 # [ 78.766509] nix-grpc-daemon[782]: ts=2026-09-22T13:46:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=0 paths=1 nar_bytes_in=5125440worker1 # [ 78.771537] nix-grpc-daemon[782]: ts=2026-09-22T13:46:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427505441worker1 # [ 78.780471] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5442worker1 # [ 78.786261] nix-daemon[848]: reaped child process 1139, status = succeeded5443worker1 # [ 78.790374] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5444worker1 # [ 79.828382] nix-daemon[848]: reaped child process 1153, status = succeeded5445worker1 # [ 79.900946] nix-daemon[848]: accepted connection from pid 1169, user nix-grpc-daemon (trusted)5446worker1 # [ 79.911802] nix-daemon[848]: reaped child process 1172, status = succeeded5447lb # [ 79.918322] niks3-server[1008]: 2026/09/22 13:46:35 INFO Received uploads request method=POST path=/api/pending_closures5448worker1 # [ 79.929379] nix-grpc-daemon[862]: time=2026-09-22T13:46:34.201Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5449worker1 # [ 79.930877] nix-grpc-daemon[862]: time=2026-09-22T13:46:34.203Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"5450lb # [ 79.954073] niks3-server[1008]: 2026/09/22 13:46:35 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst5451lb # [ 79.968847] niks3-server[1008]: 2026/09/22 13:46:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5452worker1 # [ 79.983250] nix-grpc-daemon[862]: time=2026-09-22T13:46:34.255Z level=INFO msg="Uploading 1 narinfos"5453lb # [ 79.980533] niks3-server[1008]: 2026/09/22 13:46:35 INFO Signed narinfos id=22 count=15454lb # [ 79.984490] niks3-server[1008]: 2026/09/22 13:46:35 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls5455lb # [ 79.988890] niks3-server[1008]: 2026/09/22 13:46:35 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv5456lb # [ 79.999079] niks3-server[1008]: 2026/09/22 13:46:35 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5457lb # [ 80.003329] niks3-server[1008]: 2026/09/22 13:46:35 INFO Completed upload id=225458worker1 # [ 80.006437] nix-grpc-daemon[862]: time=2026-09-22T13:46:34.279Z level=INFO msg="Upload complete. (209ms)"5459lb # [ 80.006072] niks3-server[1008]: 2026/09/22 13:46:35 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo5460worker1 # [ 80.012741] nix-grpc-daemon[782]: ts=2026-09-22T13:46:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=12 outputs=15461worker2 # [ 80.027814] nix-grpc-daemon[784]: ts=2026-09-22T13:46:35Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:364805462worker1 # [ 80.026881] nix-daemon[848]: reaped child process 1151, status = succeeded5463worker2 # [ 80.037490] nix-grpc-daemon[784]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...5464worker2 # [ 80.066720] nix-grpc-daemon[784]: ts=2026-09-22T13:46:35Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:364805465worker2 # [ 80.071946] nix-grpc-daemon[784]: ts=2026-09-22T13:46:35Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 paths=1 nar_bytes_out=1285466worker1 # [ 80.272103] nix-grpc-daemon[782]: ts=2026-09-22T13:46:34Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:427505467worker1 # [ 80.277821] nix-grpc-daemon[782]: ts=2026-09-22T13:46:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427505468worker1 # [ 80.283142] nix-grpc-daemon[862]: time=2026-09-22T13:46:34.554Z level=INFO msg="All 1 paths already cached"5469worker2 # [ 80.302407] nix-grpc-daemon[784]: ts=2026-09-22T13:46:35Z level=debug event=want client=12 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=05470worker2 # [ 80.307329] nix-grpc-daemon[784]: ts=2026-09-22T13:46:35Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=15471worker1 # [ 80.300878] nix-grpc-daemon[782]: ts=2026-09-22T13:46:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427505472worker1 # [ 80.312807] nix-grpc-daemon[782]: ts=2026-09-22T13:46:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427505473worker1 # [ 80.325067] nix-grpc-daemon[862]: time=2026-09-22T13:46:34.597Z level=INFO msg="All 1 paths already cached"5474worker1 # [ 80.332167] nix-grpc-daemon[782]: ts=2026-09-22T13:46:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:427505475worker1 # [ 80.339888] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5476worker1 # [ 80.442212] nix-daemon[848]: accepted connection from pid 1176, user nix-grpc-daemon (trusted)5477worker1 # [ 80.455043] nix-daemon[848]: reaped child process 1179, status = succeeded5478worker1 # [ 80.534111] nix-daemon[848]: accepted connection from pid 1181, user nix-grpc-daemon (trusted)5479worker1 # [ 80.546218] nix-daemon[848]: reaped child process 1184, status = succeeded5480lb # [ 80.549846] niks3-server[1008]: 2026/09/22 13:46:35 INFO Received uploads request method=POST path=/api/pending_closures5481worker1 # [ 80.557389] nix-grpc-daemon[862]: time=2026-09-22T13:46:34.829Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5482worker1 # [ 80.558771] nix-grpc-daemon[862]: time=2026-09-22T13:46:34.829Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"5483lb # [ 80.562671] niks3-server[1008]: 2026/09/22 13:46:35 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst5484lb # [ 80.567939] niks3-server[1008]: 2026/09/22 13:46:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5485lb # [ 80.570411] niks3-server[1008]: 2026/09/22 13:46:35 INFO Signed narinfos id=23 count=15486worker1 # [ 80.573302] nix-grpc-daemon[862]: time=2026-09-22T13:46:34.845Z level=INFO msg="Uploading 1 narinfos"5487lb # [ 80.574602] niks3-server[1008]: 2026/09/22 13:46:35 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls5488lb # [ 80.579255] niks3-server[1008]: 2026/09/22 13:46:35 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5489lb # [ 80.583590] niks3-server[1008]: 2026/09/22 13:46:35 INFO Completed upload id=235490worker1 # [ 80.586496] nix-grpc-daemon[862]: time=2026-09-22T13:46:34.859Z level=INFO msg="Upload complete. (233ms)"5491worker1 # [ 80.588485] nix-grpc-daemon[782]: ts=2026-09-22T13:46:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=0 paths=1 nar_bytes_in=6245492lb # [ 80.586666] niks3-server[1008]: 2026/09/22 13:46:35 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo5493worker1 # [ 80.592161] nix-grpc-daemon[782]: ts=2026-09-22T13:46:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427505494worker1 # [ 80.598432] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5495worker1 # [ 80.600638] nix-daemon[848]: reaped child process 1174, status = succeeded5496worker1 # [ 80.605629] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5497worker1 # [ 81.840965] nix-daemon[848]: reaped child process 1188, status = succeeded5498worker1 # [ 81.925408] nix-daemon[848]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)5499worker1 # [ 81.937500] nix-daemon[848]: reaped child process 1207, status = succeeded5500lb # [ 81.942425] niks3-server[1008]: 2026/09/22 13:46:37 INFO Received uploads request method=POST path=/api/pending_closures5501worker1 # [ 81.950190] nix-grpc-daemon[862]: time=2026-09-22T13:46:36.222Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5502worker1 # [ 81.951669] nix-grpc-daemon[862]: time=2026-09-22T13:46:36.224Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"5503lb # [ 81.967238] niks3-server[1008]: 2026/09/22 13:46:37 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv5504lb # [ 81.971900] niks3-server[1008]: 2026/09/22 13:46:37 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst5505lb # [ 81.975272] niks3-server[1008]: 2026/09/22 13:46:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5506lb # [ 81.977670] niks3-server[1008]: 2026/09/22 13:46:37 INFO Signed narinfos id=24 count=15507worker1 # [ 81.980599] nix-grpc-daemon[862]: time=2026-09-22T13:46:36.253Z level=INFO msg="Uploading 1 narinfos"5508lb # [ 81.980940] niks3-server[1008]: 2026/09/22 13:46:37 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls5509lb # [ 81.985823] niks3-server[1008]: 2026/09/22 13:46:37 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5510lb # [ 81.989239] niks3-server[1008]: 2026/09/22 13:46:37 INFO Completed upload id=245511worker1 # [ 81.992216] nix-grpc-daemon[862]: time=2026-09-22T13:46:36.264Z level=INFO msg="Upload complete. (179ms)"5512lb # [ 81.992341] niks3-server[1008]: 2026/09/22 13:46:37 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo5513worker1 # [ 81.995604] nix-grpc-daemon[782]: ts=2026-09-22T13:46:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=13 outputs=15514worker1 # [ 82.001124] nix-grpc-daemon[782]: ts=2026-09-22T13:46:36Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:427505515worker1 # [ 82.006088] nix-daemon[848]: reaped child process 1186, status = succeeded5516worker1 # [ 82.011268] nix-grpc-daemon[782]: ts=2026-09-22T13:46:36Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:427505517worker1 # [ 82.013683] nix-grpc-daemon[782]: ts=2026-09-22T13:46:36Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=0 paths=1 nar_bytes_out=1765518worker2 # [ 82.232876] nix-grpc-daemon[784]: ts=2026-09-22T13:46:37Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:364805519worker1 # [ 82.226748] nix-grpc-daemon[782]: ts=2026-09-22T13:46:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427505520worker1 # [ 82.235427] nix-grpc-daemon[862]: time=2026-09-22T13:46:36.508Z level=INFO msg="All 2 paths already cached"5521worker2 # [ 82.258936] nix-grpc-daemon[784]: ts=2026-09-22T13:46:37Z level=debug event=want client=13 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05522worker2 # [ 82.263939] nix-grpc-daemon[784]: ts=2026-09-22T13:46:37Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=15523worker1 # [ 82.257390] nix-grpc-daemon[782]: ts=2026-09-22T13:46:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427505524worker1 # [ 82.269724] nix-grpc-daemon[782]: ts=2026-09-22T13:46:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427505525worker1 # [ 82.282404] nix-grpc-daemon[862]: time=2026-09-22T13:46:36.555Z level=INFO msg="All 2 paths already cached"5526worker1 # [ 82.289863] nix-grpc-daemon[782]: ts=2026-09-22T13:46:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:427505527worker1 # [ 82.299347] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5528worker1 # [ 82.391372] nix-daemon[848]: accepted connection from pid 1211, user nix-grpc-daemon (trusted)5529worker1 # [ 82.406073] nix-daemon[848]: reaped child process 1214, status = succeeded5530worker1 # [ 82.476939] nix-daemon[848]: accepted connection from pid 1216, user nix-grpc-daemon (trusted)5531worker1 # [ 82.489576] nix-daemon[848]: reaped child process 1219, status = succeeded5532lb # [ 82.491883] niks3-server[1008]: 2026/09/22 13:46:37 INFO Received uploads request method=POST path=/api/pending_closures5533worker1 # [ 82.502940] nix-grpc-daemon[862]: time=2026-09-22T13:46:36.775Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5534worker1 # [ 82.504311] nix-grpc-daemon[862]: time=2026-09-22T13:46:36.775Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5535lb # [ 82.518389] niks3-server[1008]: 2026/09/22 13:46:37 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5536lb # [ 82.529506] niks3-server[1008]: 2026/09/22 13:46:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5537lb # [ 82.535163] niks3-server[1008]: 2026/09/22 13:46:37 INFO Signed narinfos id=25 count=15538worker1 # [ 82.541055] nix-grpc-daemon[862]: time=2026-09-22T13:46:36.813Z level=INFO msg="Uploading 1 narinfos"5539lb # [ 82.546547] niks3-server[1008]: 2026/09/22 13:46:37 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5540lb # [ 82.555891] niks3-server[1008]: 2026/09/22 13:46:37 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5541worker1 # [ 82.563414] nix-grpc-daemon[862]: time=2026-09-22T13:46:36.836Z level=INFO msg="Upload complete. (251ms)"5542lb # [ 82.560721] niks3-server[1008]: 2026/09/22 13:46:37 INFO Completed upload id=255543worker1 # [ 82.565577] nix-grpc-daemon[782]: ts=2026-09-22T13:46:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=0 paths=1 nar_bytes_in=7525544worker1 # [ 82.570356] nix-grpc-daemon[782]: ts=2026-09-22T13:46:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427505545lb # [ 82.570593] niks3-server[1008]: 2026/09/22 13:46:37 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo5546worker1 # [ 82.580405] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5547worker1 # [ 82.585621] nix-daemon[848]: reaped child process 1209, status = succeeded5548worker1 # [ 82.590714] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5549worker1 # [ 83.427725] nix-daemon[848]: reaped child process 1223, status = succeeded5550worker1 # [ 83.505297] nix-daemon[848]: accepted connection from pid 1239, user nix-grpc-daemon (trusted)5551worker1 # [ 83.518546] nix-daemon[848]: reaped child process 1242, status = succeeded5552lb # [ 83.522379] niks3-server[1008]: 2026/09/22 13:46:38 INFO Received uploads request method=POST path=/api/pending_closures5553worker1 # [ 83.532781] nix-grpc-daemon[862]: time=2026-09-22T13:46:37.805Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5554worker1 # [ 83.535163] nix-grpc-daemon[862]: time=2026-09-22T13:46:37.807Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5555lb # [ 83.557177] niks3-server[1008]: 2026/09/22 13:46:38 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5556lb # [ 83.570247] niks3-server[1008]: 2026/09/22 13:46:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5557lb # [ 83.575531] niks3-server[1008]: 2026/09/22 13:46:38 INFO Signed narinfos id=26 count=15558worker1 # [ 83.584219] nix-grpc-daemon[862]: time=2026-09-22T13:46:37.856Z level=INFO msg="Uploading 1 narinfos"5559lb # [ 83.581089] niks3-server[1008]: 2026/09/22 13:46:38 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5560lb # [ 83.587483] niks3-server[1008]: 2026/09/22 13:46:38 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5561lb # [ 83.596887] niks3-server[1008]: 2026/09/22 13:46:38 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5562lb # [ 83.601084] niks3-server[1008]: 2026/09/22 13:46:38 INFO Completed upload id=265563lb # [ 83.602327] niks3-server[1008]: 2026/09/22 13:46:38 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5564worker1 # [ 83.604139] nix-grpc-daemon[862]: time=2026-09-22T13:46:37.876Z level=INFO msg="Upload complete. (207ms)"5565worker1 # [ 83.611179] nix-grpc-daemon[782]: ts=2026-09-22T13:46:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=14 outputs=15566worker1 # [ 83.625401] nix-daemon[848]: reaped child process 1221, status = succeeded5567worker1 # [ 83.655557] nix-grpc-daemon[782]: ts=2026-09-22T13:46:37Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:427505568worker2 # [ 83.683877] nix-grpc-daemon[784]: ts=2026-09-22T13:46:38Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:364805569worker2 # [ 83.700622] nix-grpc-daemon[784]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...5570worker2 # [ 83.712930] nix-grpc-daemon[784]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...5571worker2 # [ 83.725579] nix-grpc-daemon[784]: ts=2026-09-22T13:46:38Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 paths=1 nar_bytes_out=2325572client # [ 83.845739] nix-daemon[1114]: reaped child process 1116, status = succeeded5573client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag hook 2>/tmp/hook.log, in 5.69 seconds)5574client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5575client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.02 seconds)5576client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5577client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.02 seconds)5578client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5579client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.02 seconds)5580(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 5.75 seconds)5581subtest: an input only one worker has still reaches the builder5582client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only5583client # this derivation will be built:5584client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5585client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5586client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag w1only, in 0.24 seconds)5587client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5588client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.10 seconds)5589worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5590worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.12 seconds)5591worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5592worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.01 seconds)5593client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&25594client # [ 84.503928] nix-daemon[1114]: accepted connection from pid 1197, user root (trusted)5595client # this derivation will be built:5596client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5597worker2 # [ 84.603289] nix-grpc-daemon[784]: ts=2026-09-22T13:46:39Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:364805598client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5599worker1 # [ 84.597606] nix-grpc-daemon[782]: ts=2026-09-22T13:46:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427505600worker1 # [ 84.706029] nix-daemon[848]: accepted connection from pid 1249, user nix-grpc-daemon (trusted)5601worker1 # [ 84.717047] nix-daemon[848]: reaped child process 1252, status = succeeded5602lb # [ 84.722944] niks3-server[1008]: 2026/09/22 13:46:39 INFO Received uploads request method=POST path=/api/pending_closures5603worker1 # [ 84.734791] nix-grpc-daemon[862]: time=2026-09-22T13:46:39.007Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5604worker1 # [ 84.736566] nix-grpc-daemon[862]: time=2026-09-22T13:46:39.008Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5605lb # [ 84.750362] niks3-server[1008]: 2026/09/22 13:46:39 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5606lb # [ 84.761772] niks3-server[1008]: 2026/09/22 13:46:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5607worker1 # [ 84.773300] nix-grpc-daemon[862]: time=2026-09-22T13:46:39.045Z level=INFO msg="Uploading 1 narinfos"5608lb # [ 84.770390] niks3-server[1008]: 2026/09/22 13:46:39 INFO Signed narinfos id=27 count=15609lb # [ 84.776787] niks3-server[1008]: 2026/09/22 13:46:39 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5610lb # [ 84.786962] niks3-server[1008]: 2026/09/22 13:46:39 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5611lb # [ 84.794288] niks3-server[1008]: 2026/09/22 13:46:39 INFO Completed upload id=275612worker1 # [ 84.797266] nix-grpc-daemon[862]: time=2026-09-22T13:46:39.069Z level=INFO msg="Upload complete. (195ms)"5613lb # [ 84.798926] niks3-server[1008]: 2026/09/22 13:46:39 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5614worker2 # [ 84.826338] nix-grpc-daemon[784]: ts=2026-09-22T13:46:39Z level=debug event=want client=14 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05615worker2 # [ 84.831588] nix-grpc-daemon[784]: ts=2026-09-22T13:46:39Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=15616worker2 # [ 84.836705] nix-grpc-daemon[784]: ts=2026-09-22T13:46:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364805617worker2 # [ 84.848799] nix-grpc-daemon[784]: ts=2026-09-22T13:46:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:364805618client # copying 2 paths to 'lb:50051'...5619client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5620worker2 # [ 84.865300] nix-grpc-daemon[784]: ts=2026-09-22T13:46:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:364805621client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5622worker2 # [ 84.879252] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5623worker2 # [ 84.988981] nix-daemon[870]: accepted connection from pid 1180, user nix-grpc-daemon (trusted)5624worker2 # [ 85.004301] nix-daemon[870]: reaped child process 1183, status = succeeded5625worker2 # [ 85.090256] nix-daemon[870]: accepted connection from pid 1185, user nix-grpc-daemon (trusted)5626worker2 # [ 85.104320] nix-daemon[870]: reaped child process 1188, status = succeeded5627lb # [ 85.099728] niks3-server[1008]: 2026/09/22 13:46:40 INFO Received uploads request method=POST path=/api/pending_closures5628lb # [ 85.108327] niks3-server[1008]: 2026/09/22 13:46:40 INFO Received uploads request method=POST path=/api/pending_closures5629worker2 # [ 85.129332] nix-grpc-daemon[922]: time=2026-09-22T13:46:40.191Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5630worker2 # [ 85.130679] nix-grpc-daemon[922]: time=2026-09-22T13:46:40.191Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5631worker2 # [ 85.132416] nix-grpc-daemon[922]: time=2026-09-22T13:46:40.191Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5632lb # [ 85.142717] niks3-server[1008]: 2026/09/22 13:46:40 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5633lb # [ 85.158685] niks3-server[1008]: 2026/09/22 13:46:40 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5634lb # [ 85.165562] niks3-server[1008]: 2026/09/22 13:46:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5635lb # [ 85.169315] niks3-server[1008]: 2026/09/22 13:46:40 INFO Signed narinfos id=28 count=15636lb # [ 85.174095] niks3-server[1008]: 2026/09/22 13:46:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5637lb # [ 85.175619] niks3-server[1008]: 2026/09/22 13:46:40 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5638lb # [ 85.177876] niks3-server[1008]: 2026/09/22 13:46:40 INFO Signed narinfos id=29 count=25639worker2 # [ 85.190405] nix-grpc-daemon[922]: time=2026-09-22T13:46:40.252Z level=INFO msg="Uploading 3 narinfos"5640lb # [ 85.183892] niks3-server[1008]: 2026/09/22 13:46:40 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5641lb # [ 85.194115] niks3-server[1008]: 2026/09/22 13:46:40 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5642lb # [ 85.197229] niks3-server[1008]: 2026/09/22 13:46:40 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5643lb # [ 85.200137] niks3-server[1008]: 2026/09/22 13:46:40 INFO Completed upload id=285644lb # [ 85.201669] niks3-server[1008]: 2026/09/22 13:46:40 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5645lb # [ 85.203981] niks3-server[1008]: 2026/09/22 13:46:40 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5646lb # [ 85.207837] niks3-server[1008]: 2026/09/22 13:46:40 INFO Completed upload id=295647lb # [ 85.209198] niks3-server[1008]: 2026/09/22 13:46:40 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5648worker2 # [ 85.219854] nix-grpc-daemon[922]: time=2026-09-22T13:46:40.282Z level=INFO msg="Upload complete. (325ms)"5649worker2 # [ 85.225466] nix-grpc-daemon[784]: ts=2026-09-22T13:46:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 paths=2 nar_bytes_in=9765650worker2 # [ 85.231878] nix-grpc-daemon[784]: ts=2026-09-22T13:46:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364805651worker2 # [ 85.245115] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5652worker2 # [ 85.253509] nix-daemon[870]: reaped child process 1178, status = succeeded5653worker2 # [ 85.262368] nix-grpc-daemon[784]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5654worker2 # [ 85.269206] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5655client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5656client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5657client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5658lb # [ 85.585281] envoy[917]: conn peer=[2001:db8:1::1]:34030 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2969 tx=790 ms=41945659lb # [ 85.588888] envoy[917]: conn peer=[2001:db8:1::1]:34042 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1589 tx=1595 ms=47915660lb # [ 85.592436] envoy[917]: conn peer=[2001:db8:1::1]:48672 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3407 tx=1923 ms=16735661lb # [ 85.596180] envoy[917]: conn peer=[2001:db8:1::1]:48676 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=333 ms=635662lb # [ 85.600203] envoy[917]: conn peer=[2001:db8:1::1]:48686 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3863 tx=2139 ms=17935663lb # [ 85.604412] envoy[917]: conn peer=[2001:db8:1::1]:48700 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=435 tx=378 ms=515664lb # [ 85.607937] envoy[917]: conn peer=[2001:db8:1::1]:48708 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4311 tx=2371 ms=15445665lb # [ 85.611446] envoy[917]: conn peer=[2001:db8:1::1]:57368 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=411 ms=965666worker2 # [ 85.682297] nix-daemon[870]: reaped child process 1193, status = succeeded5667worker2 # [ 85.760691] nix-daemon[870]: accepted connection from pid 1209, user nix-grpc-daemon (trusted)5668worker2 # [ 85.773885] nix-daemon[870]: reaped child process 1212, status = succeeded5669lb # [ 85.771672] niks3-server[1008]: 2026/09/22 13:46:40 INFO Received uploads request method=POST path=/api/pending_closures5670worker2 # [ 85.790842] nix-grpc-daemon[922]: time=2026-09-22T13:46:40.853Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5671lb # [ 85.801418] niks3-server[1008]: 2026/09/22 13:46:40 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5672lb # [ 85.807124] niks3-server[1008]: 2026/09/22 13:46:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5673lb # [ 85.812456] niks3-server[1008]: 2026/09/22 13:46:40 INFO Signed narinfos id=30 count=15674worker2 # [ 85.827393] nix-grpc-daemon[922]: time=2026-09-22T13:46:40.890Z level=INFO msg="Uploading 1 narinfos"5675lb # [ 85.828272] niks3-server[1008]: 2026/09/22 13:46:40 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5676lb # [ 85.833291] niks3-server[1008]: 2026/09/22 13:46:40 INFO Completed upload id=305677worker2 # [ 85.845211] nix-grpc-daemon[922]: time=2026-09-22T13:46:40.907Z level=INFO msg="Upload complete. (170ms)"5678lb # [ 85.840807] niks3-server[1008]: 2026/09/22 13:46:40 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5679worker2 # [ 85.850113] nix-grpc-daemon[784]: ts=2026-09-22T13:46:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=15 outputs=15680lb # [ 85.843639] niks3-server[1008]: 2026/09/22 13:46:40 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5681client # copying 1 paths...5682worker2 # [ 85.860352] nix-daemon[870]: reaped child process 1190, status = succeeded5683worker1 # [ 85.851300] nix-grpc-daemon[782]: ts=2026-09-22T13:46:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:427505684worker1 # [ 85.859833] nix-grpc-daemon[782]: copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'http://lb:5751'...5685client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5686worker1 # [ 85.884243] nix-grpc-daemon[782]: ts=2026-09-22T13:46:40Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:427505687worker1 # [ 85.889172] nix-grpc-daemon[782]: ts=2026-09-22T13:46:40Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=0 paths=1 nar_bytes_out=1205688client # [ 86.025355] nix-daemon[1114]: reaped child process 1200, status = succeeded5689client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt a >&2, in 1.64 seconds)5690client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&25691client # [ 86.127906] nix-daemon[1114]: accepted connection from pid 1219, user root (trusted)5692client # this derivation will be built:5693client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5694worker1 # [ 86.219653] nix-grpc-daemon[782]: ts=2026-09-22T13:46:40Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:427505695client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5696worker2 # [ 86.231943] nix-grpc-daemon[784]: ts=2026-09-22T13:46:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:364805697worker2 # [ 86.240199] nix-grpc-daemon[922]: time=2026-09-22T13:46:41.302Z level=INFO msg="All 1 paths already cached"5698worker2 # [ 86.251883] nix-grpc-daemon[784]: ts=2026-09-22T13:46:41Z level=debug event=want client=15 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05699worker2 # [ 86.256801] nix-grpc-daemon[784]: ts=2026-09-22T13:46:41Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15700worker1 # [ 86.249705] nix-grpc-daemon[782]: ts=2026-09-22T13:46:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427505701worker1 # [ 86.259377] nix-grpc-daemon[782]: ts=2026-09-22T13:46:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427505702client # copying 1 paths to 'lb:50051'...5703client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5704worker1 # [ 86.270449] nix-grpc-daemon[782]: ts=2026-09-22T13:46:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:427505705worker1 # [ 86.281130] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5706worker1 # [ 86.296539] nix-grpc-daemon[782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 55 ms (attempt 1/5)5707worker1 # [ 86.358763] nix-grpc-daemon[782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 48 ms (attempt 2/5)5708worker1 # [ 86.408708] nix-grpc-daemon[782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 153 ms (attempt 3/5)5709worker1 # [ 86.563817] nix-grpc-daemon[782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 262 ms (attempt 4/5)5710worker1 # [ 86.828174] nix-grpc-daemon[782]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5711worker1 # [ 86.831089] nix-grpc-daemon[782]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5712worker1 # [ 86.836615] nix-grpc-daemon[782]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5713worker1 # [ 86.939579] nix-daemon[848]: accepted connection from pid 1262, user nix-grpc-daemon (trusted)5714worker1 # [ 86.955900] nix-daemon[848]: reaped child process 1265, status = succeeded5715worker1 # [ 87.030581] nix-daemon[848]: accepted connection from pid 1267, user nix-grpc-daemon (trusted)5716worker1 # [ 87.045455] nix-daemon[848]: reaped child process 1270, status = succeeded5717lb # [ 87.049244] niks3-server[1008]: 2026/09/22 13:46:42 INFO Received uploads request method=POST path=/api/pending_closures5718worker1 # [ 87.059884] nix-grpc-daemon[862]: time=2026-09-22T13:46:41.332Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5719worker1 # [ 87.061304] nix-grpc-daemon[862]: time=2026-09-22T13:46:41.334Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5720lb # [ 87.082719] niks3-server[1008]: 2026/09/22 13:46:42 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5721lb # [ 87.091895] niks3-server[1008]: 2026/09/22 13:46:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5722lb # [ 87.096330] niks3-server[1008]: 2026/09/22 13:46:42 INFO Signed narinfos id=31 count=15723worker1 # [ 87.102112] nix-grpc-daemon[862]: time=2026-09-22T13:46:41.374Z level=INFO msg="Uploading 1 narinfos"5724lb # [ 87.102231] niks3-server[1008]: 2026/09/22 13:46:42 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5725lb # [ 87.115392] niks3-server[1008]: 2026/09/22 13:46:42 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5726lb # [ 87.123769] niks3-server[1008]: 2026/09/22 13:46:42 INFO Completed upload id=315727worker1 # [ 87.126133] nix-grpc-daemon[862]: time=2026-09-22T13:46:41.398Z level=INFO msg="Upload complete. (273ms)"5728lb # [ 87.125668] niks3-server[1008]: 2026/09/22 13:46:42 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5729worker1 # [ 87.130984] nix-grpc-daemon[782]: ts=2026-09-22T13:46:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=0 paths=1 nar_bytes_in=5605730worker1 # [ 87.136491] nix-grpc-daemon[782]: ts=2026-09-22T13:46:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427505731worker1 # [ 87.144204] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5732worker1 # [ 87.152329] nix-daemon[848]: reaped child process 1255, status = succeeded5733worker1 # [ 87.156319] nix-daemon[848]: accepted connection from pid 782, user nix-grpc-daemon (trusted)5734client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5735client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5736client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5737worker1 # [ 87.597239] nix-daemon[848]: reaped child process 1274, status = succeeded5738worker1 # [ 87.666896] nix-daemon[848]: accepted connection from pid 1290, user nix-grpc-daemon (trusted)5739worker1 # [ 87.678192] nix-daemon[848]: reaped child process 1293, status = succeeded5740lb # [ 87.681367] niks3-server[1008]: 2026/09/22 13:46:42 INFO Received uploads request method=POST path=/api/pending_closures5741worker1 # [ 87.691728] nix-grpc-daemon[862]: time=2026-09-22T13:46:41.964Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5742lb # [ 87.711996] niks3-server[1008]: 2026/09/22 13:46:42 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5743lb # [ 87.717830] niks3-server[1008]: 2026/09/22 13:46:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5744lb # [ 87.725222] niks3-server[1008]: 2026/09/22 13:46:42 INFO Signed narinfos id=32 count=15745worker1 # [ 87.730864] nix-grpc-daemon[862]: time=2026-09-22T13:46:42.003Z level=INFO msg="Uploading 1 narinfos"5746lb # [ 87.732096] niks3-server[1008]: 2026/09/22 13:46:42 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5747lb # [ 87.745476] niks3-server[1008]: 2026/09/22 13:46:42 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5748lb # [ 87.749619] niks3-server[1008]: 2026/09/22 13:46:42 INFO Completed upload id=325749worker1 # [ 87.752480] nix-grpc-daemon[862]: time=2026-09-22T13:46:42.025Z level=INFO msg="Upload complete. (190ms)"5750lb # [ 87.751186] niks3-server[1008]: 2026/09/22 13:46:42 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5751worker1 # [ 87.754709] nix-grpc-daemon[782]: ts=2026-09-22T13:46:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42750 duration_s=0 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=16 outputs=15752client # copying 1 paths...5753worker1 # [ 87.761935] nix-daemon[848]: reaped child process 1272, status = succeeded5754worker2 # [ 87.771917] nix-grpc-daemon[784]: ts=2026-09-22T13:46:42Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:364805755worker2 # [ 87.780381] nix-grpc-daemon[784]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5756client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5757worker2 # [ 87.804674] nix-grpc-daemon[784]: ts=2026-09-22T13:46:42Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:364805758worker2 # [ 87.809633] nix-grpc-daemon[784]: ts=2026-09-22T13:46:42Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 paths=1 nar_bytes_out=1205759client # [ 87.933835] nix-daemon[1114]: reaped child process 1222, status = succeeded5760client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt b >&2, in 1.91 seconds)5761client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&25762client # [ 88.035102] nix-daemon[1114]: accepted connection from pid 1241, user root (trusted)5763client # this derivation will be built:5764client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5765worker1 # [ 88.122981] nix-grpc-daemon[782]: ts=2026-09-22T13:46:42Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:427505766client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux'...5767worker1 # [ 88.128695] nix-grpc-daemon[782]: ts=2026-09-22T13:46:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427505768worker1 # [ 88.133629] nix-grpc-daemon[862]: time=2026-09-22T13:46:42.405Z level=INFO msg="All 1 paths already cached"5769worker2 # [ 88.157370] nix-grpc-daemon[784]: ts=2026-09-22T13:46:43Z level=debug event=want client=16 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05770worker2 # [ 88.163918] nix-grpc-daemon[784]: ts=2026-09-22T13:46:43Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15771worker2 # [ 88.169274] nix-grpc-daemon[784]: ts=2026-09-22T13:46:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364805772worker2 # [ 88.180614] nix-grpc-daemon[784]: ts=2026-09-22T13:46:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:364805773client # copying 1 paths to 'lb:50051'...5774worker2 # [ 88.196836] nix-grpc-daemon[922]: time=2026-09-22T13:46:43.257Z level=INFO msg="All 1 paths already cached"5775client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5776worker2 # [ 88.201300] nix-grpc-daemon[784]: ts=2026-09-22T13:46:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:364805777worker2 # [ 88.213348] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5778worker2 # [ 88.314792] nix-daemon[870]: accepted connection from pid 1221, user nix-grpc-daemon (trusted)5779worker2 # [ 88.330102] nix-daemon[870]: reaped child process 1224, status = succeeded5780worker2 # [ 88.408314] nix-daemon[870]: accepted connection from pid 1226, user nix-grpc-daemon (trusted)5781worker2 # [ 88.421596] nix-daemon[870]: reaped child process 1229, status = succeeded5782lb # [ 88.417746] niks3-server[1008]: 2026/09/22 13:46:43 INFO Received uploads request method=POST path=/api/pending_closures5783worker2 # [ 88.437832] nix-grpc-daemon[922]: time=2026-09-22T13:46:43.500Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5784worker2 # [ 88.439273] nix-grpc-daemon[922]: time=2026-09-22T13:46:43.501Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5785lb # [ 88.445342] niks3-server[1008]: 2026/09/22 13:46:43 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5786lb # [ 88.457641] niks3-server[1008]: 2026/09/22 13:46:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5787lb # [ 88.462272] niks3-server[1008]: 2026/09/22 13:46:43 INFO Signed narinfos id=33 count=15788worker2 # [ 88.477179] nix-grpc-daemon[922]: time=2026-09-22T13:46:43.539Z level=INFO msg="Uploading 1 narinfos"5789lb # [ 88.472476] niks3-server[1008]: 2026/09/22 13:46:43 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5790lb # [ 88.484355] niks3-server[1008]: 2026/09/22 13:46:43 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5791lb # [ 88.490628] niks3-server[1008]: 2026/09/22 13:46:43 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5792lb # [ 88.494902] niks3-server[1008]: 2026/09/22 13:46:43 INFO Completed upload id=335793worker2 # [ 88.506740] nix-grpc-daemon[922]: time=2026-09-22T13:46:43.569Z level=INFO msg="Upload complete. (281ms)"5794worker2 # [ 88.510825] nix-grpc-daemon[784]: ts=2026-09-22T13:46:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 paths=1 nar_bytes_in=5605795worker2 # [ 88.515301] nix-grpc-daemon[784]: ts=2026-09-22T13:46:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:364805796worker2 # [ 88.523266] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5797worker2 # [ 88.529328] nix-daemon[870]: reaped child process 1219, status = succeeded5798worker2 # [ 88.533515] nix-daemon[870]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5799client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5800client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5801client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5802worker2 # [ 89.749444] nix-daemon[870]: reaped child process 1233, status = succeeded5803worker2 # [ 89.829925] nix-daemon[870]: accepted connection from pid 1249, user nix-grpc-daemon (trusted)5804worker2 # [ 89.844852] nix-daemon[870]: reaped child process 1252, status = succeeded5805lb # [ 89.842797] niks3-server[1008]: 2026/09/22 13:46:44 INFO Received uploads request method=POST path=/api/pending_closures5806worker2 # [ 89.861823] nix-grpc-daemon[922]: time=2026-09-22T13:46:44.924Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5807lb # [ 89.871228] niks3-server[1008]: 2026/09/22 13:46:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5808worker2 # [ 89.891870] nix-grpc-daemon[922]: time=2026-09-22T13:46:44.954Z level=INFO msg="Uploading 1 narinfos"5809lb # [ 89.880528] niks3-server[1008]: 2026/09/22 13:46:44 INFO Signed narinfos id=34 count=15810lb # [ 89.886882] niks3-server[1008]: 2026/09/22 13:46:45 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5811lb # [ 89.892707] niks3-server[1008]: 2026/09/22 13:46:45 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5812lb # [ 89.904558] niks3-server[1008]: 2026/09/22 13:46:45 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5813worker2 # [ 89.922782] nix-grpc-daemon[922]: time=2026-09-22T13:46:44.985Z level=INFO msg="Upload complete. (181ms)"5814lb # [ 89.911154] niks3-server[1008]: 2026/09/22 13:46:45 INFO Completed upload id=345815lb # [ 89.913132] niks3-server[1008]: 2026/09/22 13:46:45 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5816worker2 # [ 89.925717] nix-grpc-daemon[784]: ts=2026-09-22T13:46:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=17 outputs=15817client # copying 1 paths...5818worker1 # [ 89.924417] nix-grpc-daemon[782]: ts=2026-09-22T13:46:44Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:427505819worker2 # [ 89.934597] nix-daemon[870]: reaped child process 1231, status = succeeded5820worker1 # [ 89.929051] nix-grpc-daemon[782]: copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'http://lb:5751'...5821client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5822worker2 # [ 89.958837] nix-grpc-daemon[784]: ts=2026-09-22T13:46:45Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:364805823worker2 # [ 89.962938] nix-grpc-daemon[784]: ts=2026-09-22T13:46:45Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36480 duration_s=0 paths=1 nar_bytes_out=1205824client # [ 90.088127] nix-daemon[1114]: reaped child process 1244, status = succeeded5825client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.15 seconds)5826(finished: subtest: an input only one worker has still reaches the builder, in 6.18 seconds)5827subtest: upload whose reference the worker lacks is completed from the cache5828client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache5829client # this derivation will be built:5830client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5831client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5832client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A input --argstr tag viacache, in 0.24 seconds)5833client: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache5834client # this derivation will be built:5835client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5836client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5837client: (finished: must succeed: nix-build --no-out-link /nix/store/f31x0nfj3flb882ga1as2gddqp8fka8w-dep.nix -A referrer --argstr tag viacache, in 0.24 seconds)5838client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5839client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.10 seconds)5840worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5841worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.09 seconds)5842worker2 # [ 90.795462] systemd[1]: Stopping Nix worker-protocol over gRPC...5843worker2 # [ 90.846689] nix-grpc-daemon[784]: ts=2026-09-22T13:46:45Z level=info event=signal_exit5844lb # [ 91.265503] niks3-server[1008]: 2026/09/22 13:46:46 INFO lead: released remote=192.168.1.4:340945845worker1 # [ 92.273864] nix-grpc-daemon[782]: ts=2026-09-22T13:46:46Z level=info event=scheduler_restarting addr=lb:500515846worker2 # [ 92.281584] nix-grpc-daemon[784]: ts=2026-09-22T13:46:47Z level=info event=scheduler_restarting5847worker2 # [ 92.287748] nix-grpc-daemon[784]: ts=2026-09-22T13:46:47Z level=info event=scheduler_restarting addr=lb:500515848worker2 # [ 92.291607] nix-grpc-daemon[784]: ts=2026-09-22T13:46:47Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:364745849worker2 # [ 92.295259] nix-grpc-daemon[784]: ts=2026-09-22T13:46:47Z level=info event=worker_gone addr=192.168.1.4:500515850worker2 # [ 92.296678] nix-grpc-daemon[784]: ts=2026-09-22T13:46:47Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:364745851worker2 # [ 92.297977] nix-grpc-daemon[784]: ts=2026-09-22T13:46:47Z level=info event=worker_gone addr=192.168.1.3:500515852worker2 # [ 92.312458] nix-daemon[870]: reaped child process 872, status = succeeded5853worker1 # [ 92.473795] nix-grpc-daemon[782]: ts=2026-09-22T13:46:46Z level=info event=scheduler_connected addr=lb:500515854worker1 # [ 92.477659] nix-grpc-daemon[782]: ts=2026-09-22T13:46:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5855worker2 # [ 92.494254] nix-grpc-daemon[784]: ts=2026-09-22T13:46:47Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5856worker1 # [ 92.679421] nix-grpc-daemon[782]: ts=2026-09-22T13:46:46Z level=info event=scheduler_connected addr=lb:500515857worker1 # [ 92.682490] nix-grpc-daemon[782]: ts=2026-09-22T13:46:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5858worker1 # [ 92.880324] nix-grpc-daemon[782]: ts=2026-09-22T13:46:47Z level=info event=scheduler_connected addr=lb:500515859worker1 # [ 92.883948] nix-grpc-daemon[782]: ts=2026-09-22T13:46:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5860worker2 # [ 92.928824] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5861worker2 # [ 92.931802] systemd[1]: Stopped Nix worker-protocol over gRPC.5862worker2 # [ 92.936503] systemd[1]: nix-grpc-daemon.service: Consumed 2.360s CPU time over 1min 17.249s wall clock time, 64.2M memory peak, 93.5K incoming IP traffic, 259.3K outgoing IP traffic.5863worker2 # [ 92.946750] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5864worker2 # [ 92.950229] systemd[1]: Closed Nix worker-protocol over gRPC.5865worker2 # [ 92.952630] systemd[1]: nix-grpc-daemon.socket: Consumed 348us CPU time over 1min 25.325s wall clock time, 256K memory peak, 108.7K incoming IP traffic, 95.1K outgoing IP traffic.5866lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5867lb: must succeed: curl -sf localhost:9901/clusters5868lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5869lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.03 seconds)5870client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25871worker1 # [ 93.084744] nix-grpc-daemon[782]: ts=2026-09-22T13:46:47Z level=info event=scheduler_connected addr=lb:500515872worker1 # [ 93.088737] nix-grpc-daemon[782]: ts=2026-09-22T13:46:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5873worker1 # [ 93.111869] nix-grpc-daemon[782]: ts=2026-09-22T13:46:47Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:427505874worker1 # [ 93.118679] nix-grpc-daemon[782]: ts=2026-09-22T13:46:47Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:427505875client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5876client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.16 seconds)5877worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5878worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.02 seconds)5879worker2 # [ 93.212240] systemd[1]: Listening on Nix worker-protocol over gRPC.5880worker1 # [ 93.250822] systemd[1]: Stopping Nix worker-protocol over gRPC...5881worker1 # [ 93.266333] nix-grpc-daemon[782]: ts=2026-09-22T13:46:47Z level=info event=scheduler_take_over5882lb # [ 93.264152] niks3-server[1008]: 2026/09/22 13:46:48 INFO lead: acquired remote=192.168.1.3:504565883worker1 # [ 93.290182] nix-grpc-daemon[782]: ts=2026-09-22T13:46:47Z level=info event=scheduler_connected addr=lb:500515884worker1 # [ 93.293800] nix-grpc-daemon[782]: ts=2026-09-22T13:46:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5885worker1 # [ 93.494620] nix-grpc-daemon[782]: ts=2026-09-22T13:46:47Z level=info event=scheduler_connected addr=lb:500515886worker1 # [ 93.498484] nix-grpc-daemon[782]: ts=2026-09-22T13:46:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5887worker1 # [ 93.699272] nix-grpc-daemon[782]: ts=2026-09-22T13:46:47Z level=info event=scheduler_connected addr=lb:500515888worker1 # [ 93.703233] nix-grpc-daemon[782]: ts=2026-09-22T13:46:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5889worker1 # [ 93.732900] nix-grpc-daemon[782]: ts=2026-09-22T13:46:48Z level=info event=signal_exit5890worker1 # [ 93.903499] nix-grpc-daemon[782]: ts=2026-09-22T13:46:48Z level=info event=scheduler_connected addr=lb:500515891worker1 # [ 93.905267] nix-grpc-daemon[782]: ts=2026-09-22T13:46:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5892worker2 # [ 93.994556] systemd[1]: Starting scheduler-token.service...5893worker2 # [ 94.053367] systemd[1]: scheduler-token.service: Deactivated successfully.5894worker2 # [ 94.056456] systemd[1]: Finished scheduler-token.service.5895worker2 # [ 94.059748] systemd[1]: scheduler-token.service: Consumed 35ms CPU time over 63ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5896worker2 # [ 94.066576] systemd[1]: Starting Nix worker-protocol over gRPC...5897worker1 # [ 94.105758] nix-grpc-daemon[782]: ts=2026-09-22T13:46:48Z level=info event=scheduler_connected addr=lb:500515898worker1 # [ 94.107551] nix-grpc-daemon[782]: ts=2026-09-22T13:46:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5899worker2 # [ 94.144923] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:49Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5900worker2 # [ 94.146933] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:49Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5901worker2 # [ 94.150106] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:49Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15902worker2 # [ 94.154346] nix-grpc-daemon[1276]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5903worker2 # [ 94.155654] nix-grpc-daemon[1276]: E0000 00:00:1790084809.216877 1276 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5904worker2 # [ 94.161346] nix-grpc-daemon[1276]: E0000 00:00:1790084809.222145 1276 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5905worker2 # [ 94.165523] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:49Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5906worker2 # [ 94.167189] systemd[1]: Started Nix worker-protocol over gRPC.5907worker2 # [ 94.173066] nix-grpc-daemon[1276]: E0000 00:00:1790084809.235292 1318 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5908worker2 # [ 94.186718] nix-grpc-daemon[1276]: E0000 00:00:1790084809.249351 1318 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5909worker2 # [ 94.190802] nix-grpc-daemon[1276]: E0000 00:00:1790084809.251630 1318 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5910worker2 # [ 94.200749] nix-grpc-daemon[1276]: E0000 00:00:1790084809.263411 1318 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5911worker2 # [ 94.209929] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:49Z level=info event=scheduler_connected addr=lb:500515912worker2 # [ 94.211488] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5913lb # [ 94.270734] niks3-server[1008]: 2026/09/22 13:46:49 INFO lead: released remote=192.168.1.3:504565914worker1 # [ 94.308412] nix-grpc-daemon[782]: ts=2026-09-22T13:46:48Z level=info event=scheduler_connected addr=lb:500515915worker1 # [ 94.310166] nix-grpc-daemon[782]: ts=2026-09-22T13:46:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5916worker2 # [ 94.412948] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:49Z level=info event=scheduler_connected addr=lb:500515917worker2 # [ 94.417211] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5918worker1 # [ 94.511081] nix-grpc-daemon[782]: ts=2026-09-22T13:46:48Z level=info event=scheduler_connected addr=lb:500515919worker1 # [ 94.515190] nix-grpc-daemon[782]: ts=2026-09-22T13:46:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5920worker2 # [ 94.618428] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:49Z level=info event=scheduler_connected addr=lb:500515921worker2 # [ 94.622405] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5922worker1 # [ 94.715942] nix-grpc-daemon[782]: ts=2026-09-22T13:46:48Z level=info event=scheduler_connected addr=lb:500515923worker1 # [ 94.719979] nix-grpc-daemon[782]: ts=2026-09-22T13:46:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5924worker2 # [ 94.823543] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:49Z level=info event=scheduler_connected addr=lb:500515925worker2 # [ 94.827499] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5926worker1 # [ 94.920575] nix-grpc-daemon[782]: ts=2026-09-22T13:46:49Z level=info event=scheduler_connected addr=lb:500515927worker1 # [ 94.924978] nix-grpc-daemon[782]: ts=2026-09-22T13:46:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5928worker2 # [ 95.028771] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:50Z level=info event=scheduler_connected addr=lb:500515929worker2 # [ 95.032566] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5930worker1 # [ 95.125915] nix-grpc-daemon[782]: ts=2026-09-22T13:46:49Z level=info event=scheduler_connected addr=lb:500515931worker1 # [ 95.130604] nix-grpc-daemon[782]: ts=2026-09-22T13:46:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5932worker2 # [ 95.234845] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:50Z level=info event=scheduler_connected addr=lb:500515933worker2 # [ 95.237940] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5934worker1 # [ 95.271670] nix-grpc-daemon[782]: ts=2026-09-22T13:46:49Z level=info event=scheduler_restarting5935worker1 # [ 95.295641] nix-daemon[848]: reaped child process 850, status = succeeded5936worker1 # [ 95.343859] nix-grpc-daemon[782]: ts=2026-09-22T13:46:49Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5937worker2 # [ 95.435840] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:50Z level=info event=scheduler_connected addr=lb:500515938worker2 # [ 95.439661] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5939lb # [ 95.586639] envoy[917]: conn peer=[2001:db8:1::1]:57380 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=565940lb # [ 95.589286] envoy[917]: conn peer=[2001:db8:1::1]:57372 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3833 tx=2029 ms=13475941lb # [ 95.593445] envoy[917]: conn peer=[2001:db8:1::1]:57386 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3698 tx=2083 ms=16305942lb # [ 95.597822] envoy[917]: conn peer=[2001:db8:1::1]:57390 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=565943lb # [ 95.602277] envoy[917]: conn peer=[2001:db8:1::1]:57404 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3756 tx=2123 ms=18835944lb # [ 95.607255] envoy[917]: conn peer=[2001:db8:1::1]:57414 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=575945lb # [ 95.611439] envoy[917]: conn peer=[2001:db8:1::4]:34426 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=75947 tx=3750 ms=766095946lb # [ 95.615735] envoy[917]: conn peer=[2001:db8:1::1]:33872 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=519 ms=205947lb # [ 95.619758] envoy[917]: conn peer=[2001:db8:1::3]:38940 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=22950 tx=4449 ms=795735948worker2 # [ 95.640940] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:50Z level=info event=scheduler_connected addr=lb:500515949worker2 # [ 95.644989] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5950worker2 # [ 95.846565] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:50Z level=info event=scheduler_connected addr=lb:500515951worker2 # [ 95.850787] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5952worker1 # [ 95.864224] nix-grpc-daemon[782]: E0000 00:00:1790084810.136809 827 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true5953worker1 # [ 95.866473] nix-grpc-daemon[782]: E0000 00:00:1790084810.139032 827 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5954worker2 # [ 96.051530] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:51Z level=info event=scheduler_connected addr=lb:500515955worker2 # [ 96.053578] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5956worker2 # [ 96.254803] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:51Z level=info event=scheduler_connected addr=lb:500515957worker2 # [ 96.256600] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5958worker2 # [ 96.457472] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:51Z level=info event=scheduler_connected addr=lb:500515959worker2 # [ 96.459413] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5960worker2 # [ 96.660186] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:51Z level=info event=scheduler_connected addr=lb:500515961worker2 # [ 96.661962] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5962worker2 # [ 96.862783] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:51Z level=info event=scheduler_connected addr=lb:500515963worker2 # [ 96.864619] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5964worker1 # [ 96.888852] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5965worker1 # [ 96.891751] systemd[1]: Stopped Nix worker-protocol over gRPC.5966worker1 # [ 96.895603] systemd[1]: nix-grpc-daemon.service: Consumed 3.024s CPU time over 1min 21.304s wall clock time, 68.5M memory peak, 109.6K incoming IP traffic, 205.7K outgoing IP traffic.5967worker1 # [ 96.907685] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5968worker1 # [ 96.911929] systemd[1]: Closed Nix worker-protocol over gRPC.5969worker1 # [ 96.915308] systemd[1]: nix-grpc-daemon.socket: Consumed 314us CPU time over 1min 28.394s wall clock time, 256K memory peak, 100.1K incoming IP traffic, 86.7K outgoing IP traffic.5970lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5971lb: must succeed: curl -sf localhost:9901/clusters5972lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5973lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.03 seconds)5974client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25975worker2 # [ 97.065420] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:52Z level=info event=scheduler_connected addr=lb:500515976worker2 # [ 97.067284] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5977worker2 # [ 97.099069] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436405978worker2 # [ 97.104804] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)5979worker2 # [ 97.112214] nix-grpc-daemon[1276]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 98 ms (attempt 1/5)5980worker2 # [ 97.214920] nix-grpc-daemon[1276]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 65 ms (attempt 2/5)5981worker2 # [ 97.269627] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:52Z level=info event=scheduler_connected addr=lb:500515982worker2 # [ 97.273312] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5983worker2 # [ 97.281592] nix-grpc-daemon[1276]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 189 ms (attempt 3/5)5984worker2 # [ 97.474096] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:52Z level=info event=scheduler_connected addr=lb:500515985worker2 # [ 97.476352] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5986worker2 # [ 97.478723] nix-grpc-daemon[1276]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 541 ms (attempt 4/5)5987worker2 # [ 97.677349] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:52Z level=info event=scheduler_connected addr=lb:500515988worker2 # [ 97.681525] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5989worker2 # [ 97.882314] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:52Z level=info event=scheduler_connected addr=lb:500515990worker2 # [ 97.884147] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5991worker2 # [ 98.024491] nix-grpc-daemon[1276]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5992worker2 # [ 98.028383] nix-grpc-daemon[1276]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5993client # copying 2 paths...5994client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5995worker2 # [ 98.046518] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436405996client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5997worker2 # [ 98.056815] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)5998worker2 # [ 98.079695] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:53Z level=info event=niks3_push_started pid=13365999worker2 # [ 98.087595] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:53Z level=info event=scheduler_connected addr=lb:500516000worker2 # [ 98.091897] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6001worker2 # [ 98.190852] nix-grpc-daemon[1336]: time=2026-09-22T13:46:53.252Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""6002worker2 # [ 98.274655] nix-daemon[870]: accepted connection from pid 1349, user nix-grpc-daemon (trusted)6003worker2 # [ 98.289106] nix-daemon[870]: reaped child process 1352, status = succeeded6004worker2 # [ 98.292361] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:53Z level=info event=scheduler_connected addr=lb:500516005worker2 # [ 98.294278] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6006lb # [ 98.286901] niks3-server[1008]: 2026/09/22 13:46:53 INFO Received uploads request method=POST path=/api/pending_closures6007lb # [ 98.292077] niks3-server[1008]: 2026/09/22 13:46:53 INFO Received uploads request method=POST path=/api/pending_closures6008worker2 # [ 98.308071] nix-grpc-daemon[1336]: time=2026-09-22T13:46:53.369Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"6009worker2 # [ 98.309700] nix-grpc-daemon[1336]: time=2026-09-22T13:46:53.369Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"6010worker2 # [ 98.313264] nix-grpc-daemon[1336]: time=2026-09-22T13:46:53.370Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"6011lb # [ 98.319390] niks3-server[1008]: 2026/09/22 13:46:53 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst6012lb # [ 98.326501] niks3-server[1008]: 2026/09/22 13:46:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign6013lb # [ 98.329941] niks3-server[1008]: 2026/09/22 13:46:53 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst6014lb # [ 98.332596] niks3-server[1008]: 2026/09/22 13:46:53 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls6015lb # [ 98.342672] niks3-server[1008]: 2026/09/22 13:46:53 INFO Signed narinfos id=36 count=26016lb # [ 98.344088] niks3-server[1008]: 2026/09/22 13:46:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign6017lb # [ 98.346823] niks3-server[1008]: 2026/09/22 13:46:53 INFO Signed narinfos id=35 count=16018worker2 # [ 98.359483] nix-grpc-daemon[1336]: time=2026-09-22T13:46:53.421Z level=INFO msg="Uploading 3 narinfos"6019lb # [ 98.353820] niks3-server[1008]: 2026/09/22 13:46:53 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls6020lb # [ 98.363428] niks3-server[1008]: 2026/09/22 13:46:53 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo6021lb # [ 98.368474] niks3-server[1008]: 2026/09/22 13:46:53 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete6022lb # [ 98.375824] niks3-server[1008]: 2026/09/22 13:46:53 INFO Completed upload id=366023lb # [ 98.377104] niks3-server[1008]: 2026/09/22 13:46:53 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete6024lb # [ 98.379509] niks3-server[1008]: 2026/09/22 13:46:53 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo6025lb # [ 98.383931] niks3-server[1008]: 2026/09/22 13:46:53 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo6026lb # [ 98.386391] niks3-server[1008]: 2026/09/22 13:46:53 INFO Completed upload id=356027worker2 # [ 98.398548] nix-grpc-daemon[1336]: time=2026-09-22T13:46:53.461Z level=INFO msg="Upload complete. (205ms)"6028worker2 # [ 98.402818] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=0 paths=2 nar_bytes_in=3126029worker2 # [ 98.414702] nix-daemon[870]: reaped child process 1334, status = succeeded6030client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.47 seconds)6031worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache6032worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.02 seconds)6033worker2 # [ 98.495391] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:53Z level=info event=scheduler_connected addr=lb:500516034worker1 # [ 98.487652] systemd[1]: Listening on Nix worker-protocol over gRPC.6035worker2 # [ 98.499217] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6036worker1 # [ 98.496171] systemd[1]: Starting Nix worker-protocol over gRPC...6037worker1 # [ 98.587624] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:52Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6038worker1 # [ 98.593227] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:52Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6039worker1 # [ 98.596194] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:52Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16040worker1 # [ 98.603206] nix-grpc-daemon[1317]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6041worker1 # [ 98.604783] nix-grpc-daemon[1317]: E0000 00:00:1790084812.865612 1317 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6042lb # [ 98.602615] niks3-server[1008]: 2026/09/22 13:46:53 INFO lead: acquired remote=192.168.1.3:541686043worker1 # [ 98.607866] nix-grpc-daemon[1317]: E0000 00:00:1790084812.870888 1317 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6044worker1 # [ 98.610168] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:52Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6045worker1 # [ 98.615161] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:52Z level=info event=scheduler_take_over6046worker1 # [ 98.616407] systemd[1]: Started Nix worker-protocol over gRPC.6047worker1 # [ 98.620139] nix-grpc-daemon[1317]: E0000 00:00:1790084812.883530 1356 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6048lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6049lb: must succeed: curl -sf localhost:9901/clusters6050worker1 # [ 98.629081] nix-grpc-daemon[1317]: E0000 00:00:1790084812.900201 1356 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6051worker1 # [ 98.631283] nix-grpc-daemon[1317]: E0000 00:00:1790084812.900470 1356 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6052worker1 # [ 98.641726] nix-grpc-daemon[1317]: E0000 00:00:1790084812.914429 1356 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6053worker1 # [ 98.652794] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:52Z level=info event=scheduler_connected addr=lb:500516054worker1 # [ 98.654388] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6055lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6056worker2 # [ 98.700488] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:53Z level=info event=scheduler_connected addr=lb:500516057worker2 # [ 98.704502] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6058worker1 # [ 98.855417] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:53Z level=info event=scheduler_connected addr=lb:500516059worker1 # [ 98.859336] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6060worker2 # [ 98.905307] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:53Z level=info event=scheduler_connected addr=lb:500516061worker1 # [ 98.906142] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:53Z level=info event=oidc_jwks provider=mock keys=16062worker1 # [ 98.909248] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:53Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:388486063worker1 # [ 98.913634] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:53Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06064worker1 # [ 99.060102] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:53Z level=info event=scheduler_connected addr=lb:500516065worker1 # [ 99.064936] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:53Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:388486066worker1 # [ 99.068613] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:53Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06067lb: must succeed: curl -sf localhost:9901/clusters6068lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6069lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.07 seconds)6070(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 9.63 seconds)6071subtest: a niks3 restart does not cost the leader its role6072lb: must succeed: curl -sf localhost:9901/clusters6073lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6074lb: must succeed: curl -sf localhost:9901/clusters6075lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6076worker1 # [ 99.761600] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:54Z level=info event=scheduler_lock_lost status=2006077lb # [ 99.757882] niks3-server[1008]: 2026/09/22 13:46:54 INFO Shutdown signal received, draining in-flight requests timeout=10s6078worker2 # [ 99.770715] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:54Z level=info event=scheduler_lock_lost status=2006079lb # [ 99.762278] niks3-server[1008]: 2026/09/22 13:46:54 INFO lead: released remote=192.168.1.3:541686080lb # [ 99.766348] systemd[1]: Stopping niks3 server...6081lb # [ 99.776793] systemd[1]: niks3.service: Deactivated successfully.6082lb # [ 99.778349] systemd[1]: Stopped niks3 server.6083lb # [ 99.779135] systemd[1]: niks3.service: Consumed 679ms CPU time over 1min 16.747s wall clock time, 28.9M memory peak, 98.2K incoming IP traffic, 140.3K outgoing IP traffic.6084lb # [ 99.786751] systemd[1]: Starting niks3 server...6085lb # [ 99.836236] niks3-server[1149]: 2026/09/22 13:46:54 goose: no migrations to run. current version: 202609200000006086lb # [ 99.842109] niks3-server[1149]: 2026/09/22 13:46:54 OK 1_commit_pending_closure.sql (4ms)6087lb # [ 99.845536] niks3-server[1149]: 2026/09/22 13:46:54 OK 2_object_stats_trigger.sql (1.85ms)6088lb # [ 99.846591] niks3-server[1149]: 2026/09/22 13:46:54 goose: up to current file version: 26089lb # [ 99.847848] niks3-server[1149]: 2026/09/22 13:46:54 INFO Loaded signing key name=farm-test-1 path=/nix/store/hk0nhypl37jvkn3gr36v3y7qx593p39i-key6090lb # [ 99.858789] niks3-server[1149]: 2026/09/22 13:46:54 INFO Read proxy enabled — serving cache objects from S36091lb # [ 99.860242] niks3-server[1149]: 2026/09/22 13:46:54 INFO Using socket-activated listener address=0.0.0.0:57516092lb # [ 99.862165] systemd[1]: Started niks3 server.6093lb # [ 99.862988] niks3-server[1149]: 2026/09/22 13:46:54 INFO systemd watchdog enabled interval=15s6094lb # [ 99.866140] niks3-server[1149]: 2026/09/22 13:46:54 INFO Starting HTTP server address=0.0.0.0:57516095lb: waiting for unit niks3.service6096lb: (finished: waiting for unit niks3.service, in 0.02 seconds)6097lb: must succeed: curl -sf localhost:9901/clusters6098lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6099worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6100worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6101lb: must succeed: curl -sf localhost:9901/clusters6102lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6103lb: must succeed: curl -sf localhost:9901/clusters6104lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6105worker1: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield6106worker1: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.05 seconds)6107client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&26108worker1 # [ 100.198171] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:388606109worker1 # [ 100.209651] nix-daemon[848]: accepted connection from pid 1317, user nix-grpc-daemon (trusted)6110client # don't know how to build these paths:6111client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6112worker1 # [ 100.224180] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:388606113worker1 # [ 100.234278] nix-grpc-daemon[1317]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 90 ms (attempt 1/5)6114worker1 # [ 100.331602] nix-grpc-daemon[1317]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 113 ms (attempt 2/5)6115worker1 # [ 100.445735] nix-grpc-daemon[1317]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 198 ms (attempt 3/5)6116worker1 # [ 100.644869] nix-grpc-daemon[1317]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 303 ms (attempt 4/5)6117lb # [ 100.762177] niks3-server[1149]: 2026/09/22 13:46:55 INFO lead: acquired remote=192.168.1.3:541746118worker1 # [ 100.949380] nix-grpc-daemon[1317]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6119worker1 # [ 100.950959] nix-grpc-daemon[1317]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6120worker1 # [ 100.969519] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:55Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=06121worker1 # [ 100.971530] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:55Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.4:50051 clients=16122worker2 # [ 101.024638] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406123worker2 # [ 101.034078] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436406124client # copying 1 paths to 'lb:50051'...6125client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...6126worker2 # [ 101.041705] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436406127worker2 # [ 101.048357] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6128worker2 # [ 101.140668] nix-daemon[870]: accepted connection from pid 1361, user nix-grpc-daemon (trusted)6129worker2 # [ 101.155875] nix-daemon[870]: reaped child process 1364, status = succeeded6130worker2 # [ 101.236897] nix-daemon[870]: accepted connection from pid 1366, user nix-grpc-daemon (trusted)6131worker2 # [ 101.252118] nix-daemon[870]: reaped child process 1369, status = succeeded6132lb # [ 101.245773] niks3-server[1149]: 2026/09/22 13:46:56 INFO Received uploads request method=POST path=/api/pending_closures6133worker2 # [ 101.269152] nix-grpc-daemon[1336]: time=2026-09-22T13:46:56.331Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6134worker2 # [ 101.271213] nix-grpc-daemon[1336]: time=2026-09-22T13:46:56.333Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"6135lb # [ 101.288807] niks3-server[1149]: 2026/09/22 13:46:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign6136lb # [ 101.298937] niks3-server[1149]: 2026/09/22 13:46:56 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst6137lb # [ 101.322896] niks3-server[1149]: 2026/09/22 13:46:56 INFO Signed narinfos id=37 count=16138worker2 # [ 101.336773] nix-grpc-daemon[1336]: time=2026-09-22T13:46:56.399Z level=INFO msg="Uploading 1 narinfos"6139lb # [ 101.324984] niks3-server[1149]: 2026/09/22 13:46:56 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls6140lb # [ 101.330233] niks3-server[1149]: 2026/09/22 13:46:56 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete6141lb # [ 101.338238] niks3-server[1149]: 2026/09/22 13:46:56 INFO Completed upload id=376142worker2 # [ 101.350395] nix-grpc-daemon[1336]: time=2026-09-22T13:46:56.412Z level=INFO msg="Upload complete. (296ms)"6143lb # [ 101.342699] niks3-server[1149]: 2026/09/22 13:46:56 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo6144worker2 # [ 101.355951] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=0 paths=1 nar_bytes_in=5366145worker2 # [ 101.362407] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406146worker2 # [ 101.376118] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6147worker2 # [ 101.384228] nix-daemon[870]: reaped child process 1359, status = succeeded6148worker2 # [ 101.391992] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6149client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...6150client # farm-a> worker2: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv6151client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6152client # farm-a> LOG-farm-a-after-niks3-restart6153worker2 # [ 102.421886] nix-daemon[870]: reaped child process 1373, status = succeeded6154worker2 # [ 102.503113] nix-daemon[870]: accepted connection from pid 1389, user nix-grpc-daemon (trusted)6155worker2 # [ 102.515505] nix-daemon[870]: reaped child process 1392, status = succeeded6156lb # [ 102.510797] niks3-server[1149]: 2026/09/22 13:46:57 INFO Received uploads request method=POST path=/api/pending_closures6157worker2 # [ 102.533498] nix-grpc-daemon[1336]: time=2026-09-22T13:46:57.595Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6158worker2 # [ 102.534941] nix-grpc-daemon[1336]: time=2026-09-22T13:46:57.597Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"6159lb # [ 102.554835] niks3-server[1149]: 2026/09/22 13:46:57 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv6160lb # [ 102.566077] niks3-server[1149]: 2026/09/22 13:46:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign6161worker2 # [ 102.587838] nix-grpc-daemon[1336]: time=2026-09-22T13:46:57.650Z level=INFO msg="Uploading 1 narinfos"6162lb # [ 102.576112] niks3-server[1149]: 2026/09/22 13:46:57 INFO Signed narinfos id=38 count=16163lb # [ 102.579087] niks3-server[1149]: 2026/09/22 13:46:57 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst6164lb # [ 102.585927] niks3-server[1149]: 2026/09/22 13:46:57 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls6165lb # [ 102.594726] niks3-server[1149]: 2026/09/22 13:46:57 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete6166lb # [ 102.598198] niks3-server[1149]: 2026/09/22 13:46:57 INFO Completed upload id=386167lb # [ 102.600472] niks3-server[1149]: 2026/09/22 13:46:57 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo6168worker2 # [ 102.611524] nix-grpc-daemon[1336]: time=2026-09-22T13:46:57.674Z level=INFO msg="Upload complete. (211ms)"6169worker2 # [ 102.617761] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=16170worker1 # [ 102.616752] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:56Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=06171worker1 # [ 102.621335] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:56Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.4:50051 clients=16172worker2 # [ 102.633768] nix-daemon[870]: reaped child process 1371, status = succeeded6173worker2 # [ 102.674304] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406174worker2 # [ 102.686420] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436406175worker2 # [ 102.699588] nix-grpc-daemon[1336]: time=2026-09-22T13:46:57.762Z level=INFO msg="All 1 paths already cached"6176client # copying 1 paths to 'lb:50051'...6177client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...6178worker2 # [ 102.708278] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436406179worker2 # [ 102.718176] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6180worker2 # [ 102.814460] nix-daemon[870]: accepted connection from pid 1396, user nix-grpc-daemon (trusted)6181worker2 # [ 102.828734] nix-daemon[870]: reaped child process 1399, status = succeeded6182worker2 # [ 102.909805] nix-daemon[870]: accepted connection from pid 1401, user nix-grpc-daemon (trusted)6183worker2 # [ 102.923319] nix-daemon[870]: reaped child process 1404, status = succeeded6184lb # [ 102.918381] niks3-server[1149]: 2026/09/22 13:46:58 INFO Received uploads request method=POST path=/api/pending_closures6185worker2 # [ 102.938682] nix-grpc-daemon[1336]: time=2026-09-22T13:46:58.001Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6186worker2 # [ 102.940078] nix-grpc-daemon[1336]: time=2026-09-22T13:46:58.001Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"6187lb # [ 102.945829] niks3-server[1149]: 2026/09/22 13:46:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6188worker2 # [ 102.964172] nix-grpc-daemon[1336]: time=2026-09-22T13:46:58.026Z level=INFO msg="Uploading 1 narinfos"6189lb # [ 102.952700] niks3-server[1149]: 2026/09/22 13:46:58 INFO Signed narinfos id=39 count=16190lb # [ 102.962483] niks3-server[1149]: 2026/09/22 13:46:58 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst6191lb # [ 102.971856] niks3-server[1149]: 2026/09/22 13:46:58 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls6192lb # [ 102.982262] niks3-server[1149]: 2026/09/22 13:46:58 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6193worker2 # [ 103.001292] nix-grpc-daemon[1336]: time=2026-09-22T13:46:58.063Z level=INFO msg="Upload complete. (271ms)"6194lb # [ 102.989733] niks3-server[1149]: 2026/09/22 13:46:58 INFO Completed upload id=396195lb # [ 102.993027] niks3-server[1149]: 2026/09/22 13:46:58 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo6196worker2 # [ 103.003924] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=0 paths=1 nar_bytes_in=6566197worker2 # [ 103.009172] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406198worker2 # [ 103.018499] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6199worker2 # [ 103.024655] nix-daemon[870]: reaped child process 1394, status = succeeded6200client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...6201worker2 # [ 103.030412] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6202client # farm-b> worker2: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv6203client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6204client # farm-b> LOG-farm-b-after-niks3-restart6205worker2 # [ 103.871669] nix-daemon[870]: reaped child process 1408, status = succeeded6206worker2 # [ 103.946137] nix-daemon[870]: accepted connection from pid 1424, user nix-grpc-daemon (trusted)6207worker2 # [ 103.959794] nix-daemon[870]: reaped child process 1427, status = succeeded6208lb # [ 103.956271] niks3-server[1149]: 2026/09/22 13:46:59 INFO Received uploads request method=POST path=/api/pending_closures6209worker2 # [ 103.976513] nix-grpc-daemon[1336]: time=2026-09-22T13:46:59.038Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6210worker2 # [ 103.977985] nix-grpc-daemon[1336]: time=2026-09-22T13:46:59.040Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"6211lb # [ 103.987105] niks3-server[1149]: 2026/09/22 13:46:59 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv6212lb # [ 104.002272] niks3-server[1149]: 2026/09/22 13:46:59 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst6213lb # [ 104.014361] niks3-server[1149]: 2026/09/22 13:46:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6214lb # [ 104.019531] niks3-server[1149]: 2026/09/22 13:46:59 INFO Signed narinfos id=40 count=16215worker2 # [ 104.030983] nix-grpc-daemon[1336]: time=2026-09-22T13:46:59.093Z level=INFO msg="Uploading 1 narinfos"6216lb # [ 104.028144] niks3-server[1149]: 2026/09/22 13:46:59 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls6217lb # [ 104.036568] niks3-server[1149]: 2026/09/22 13:46:59 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6218lb # [ 104.039153] niks3-server[1149]: 2026/09/22 13:46:59 INFO Completed upload id=406219worker2 # [ 104.051265] nix-grpc-daemon[1336]: time=2026-09-22T13:46:59.113Z level=INFO msg="Upload complete. (214ms)"6220lb # [ 104.043619] niks3-server[1149]: 2026/09/22 13:46:59 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo6221worker2 # [ 104.057594] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=16222worker1 # [ 104.056577] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:58Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=06223worker1 # [ 104.060616] nix-grpc-daemon[1317]: ts=2026-09-22T13:46:58Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.4:50051 clients=16224worker2 # [ 104.073622] nix-daemon[870]: reaped child process 1406, status = succeeded6225worker2 # [ 104.109369] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406226worker2 # [ 104.121337] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436406227worker2 # [ 104.134595] nix-grpc-daemon[1336]: time=2026-09-22T13:46:59.197Z level=INFO msg="All 2 paths already cached"6228client # copying 1 paths to 'lb:50051'...6229client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...6230worker2 # [ 104.143583] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436406231worker2 # [ 104.153180] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6232worker2 # [ 104.252128] nix-daemon[870]: accepted connection from pid 1431, user nix-grpc-daemon (trusted)6233worker2 # [ 104.266382] nix-daemon[870]: reaped child process 1434, status = succeeded6234worker2 # [ 104.348166] nix-daemon[870]: accepted connection from pid 1436, user nix-grpc-daemon (trusted)6235worker2 # [ 104.364104] nix-daemon[870]: reaped child process 1439, status = succeeded6236lb # [ 104.356700] niks3-server[1149]: 2026/09/22 13:46:59 INFO Received uploads request method=POST path=/api/pending_closures6237worker2 # [ 104.374505] nix-grpc-daemon[1336]: time=2026-09-22T13:46:59.437Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6238worker2 # [ 104.376083] nix-grpc-daemon[1336]: time=2026-09-22T13:46:59.438Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"6239lb # [ 104.377831] niks3-server[1149]: 2026/09/22 13:46:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6240lb # [ 104.380230] niks3-server[1149]: 2026/09/22 13:46:59 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst6241lb # [ 104.382562] niks3-server[1149]: 2026/09/22 13:46:59 INFO Signed narinfos id=41 count=16242worker2 # [ 104.395472] nix-grpc-daemon[1336]: time=2026-09-22T13:46:59.458Z level=INFO msg="Uploading 1 narinfos"6243lb # [ 104.385164] niks3-server[1149]: 2026/09/22 13:46:59 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls6244lb # [ 104.390751] niks3-server[1149]: 2026/09/22 13:46:59 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6245lb # [ 104.396511] niks3-server[1149]: 2026/09/22 13:46:59 INFO Completed upload id=416246worker2 # [ 104.408315] nix-grpc-daemon[1336]: time=2026-09-22T13:46:59.471Z level=INFO msg="Upload complete. (241ms)"6247lb # [ 104.398417] niks3-server[1149]: 2026/09/22 13:46:59 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo6248worker2 # [ 104.410693] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=0 paths=1 nar_bytes_in=7846249worker2 # [ 104.414306] nix-grpc-daemon[1276]: ts=2026-09-22T13:46:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406250worker2 # [ 104.421061] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6251worker2 # [ 104.423296] nix-daemon[870]: reaped child process 1429, status = succeeded6252worker2 # [ 104.428858] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6253client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...6254client # farm-top> worker2: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6255client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6256lb # [ 105.586978] envoy[917]: conn peer=[2001:db8:1::1]:33882 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1006 tx=305 ms=13146257client # farm-top> LOG-farm-top-after-niks3-restart6258worker2 # [ 105.644889] nix-daemon[870]: reaped child process 1443, status = succeeded6259worker2 # [ 105.722510] nix-daemon[870]: accepted connection from pid 1459, user nix-grpc-daemon (trusted)6260worker2 # [ 105.737254] nix-daemon[870]: reaped child process 1462, status = succeeded6261lb # [ 105.733775] niks3-server[1149]: 2026/09/22 13:47:00 INFO Received uploads request method=POST path=/api/pending_closures6262worker2 # [ 105.755455] nix-grpc-daemon[1336]: time=2026-09-22T13:47:00.817Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6263worker2 # [ 105.756977] nix-grpc-daemon[1336]: time=2026-09-22T13:47:00.819Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"6264lb # [ 105.776644] niks3-server[1149]: 2026/09/22 13:47:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6265lb # [ 105.784073] niks3-server[1149]: 2026/09/22 13:47:00 INFO Signed narinfos id=42 count=16266worker2 # [ 105.798948] nix-grpc-daemon[1336]: time=2026-09-22T13:47:00.861Z level=INFO msg="Uploading 1 narinfos"6267lb # [ 105.795911] niks3-server[1149]: 2026/09/22 13:47:00 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv6268lb # [ 105.799917] niks3-server[1149]: 2026/09/22 13:47:00 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst6269lb # [ 105.809252] niks3-server[1149]: 2026/09/22 13:47:00 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6270lb # [ 105.816359] niks3-server[1149]: 2026/09/22 13:47:00 INFO Completed upload id=426271worker2 # [ 105.828095] nix-grpc-daemon[1336]: time=2026-09-22T13:47:00.890Z level=INFO msg="Upload complete. (192ms)"6272lb # [ 105.818396] niks3-server[1149]: 2026/09/22 13:47:00 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls6273lb # [ 105.820945] niks3-server[1149]: 2026/09/22 13:47:00 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo6274worker2 # [ 105.834797] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=16275worker2 # [ 105.848842] nix-daemon[870]: reaped child process 1441, status = succeeded6276client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart >&2, in 5.81 seconds)6277client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath6278client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag after-niks3-restart outPath, in 0.12 seconds)6279(finished: subtest: a niks3 restart does not cost the leader its role, in 6.28 seconds)6280subtest: scheduler down: the other node takes the lock and keeps it6281lb: must succeed: curl -sf localhost:9901/clusters6282lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6283lb: must succeed: curl -sf localhost:9901/clusters6284lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6285lb: must succeed: curl -sf localhost:9901/clusters6286lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6287lb: must succeed: curl -sf localhost:9901/clusters6288lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6289worker1 # [ 106.139821] systemd[1]: Stopping Nix worker-protocol over gRPC...6290worker1 # [ 106.603995] nix-grpc-daemon[1317]: ts=2026-09-22T13:47:00Z level=info event=signal_exit6291lb # [ 106.767850] niks3-server[1149]: 2026/09/22 13:47:01 INFO lead: released remote=192.168.1.3:541746292worker2 # [ 107.779684] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:02Z level=info event=scheduler_restarting addr=lb:500516293worker1 # [ 107.774387] nix-grpc-daemon[1317]: ts=2026-09-22T13:47:02Z level=info event=scheduler_restarting6294worker1 # [ 107.778378] nix-grpc-daemon[1317]: ts=2026-09-22T13:47:02Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:388486295worker1 # [ 107.781773] nix-grpc-daemon[1317]: ts=2026-09-22T13:47:02Z level=info event=worker_gone addr=192.168.1.4:500516296worker1 # [ 107.785222] nix-grpc-daemon[1317]: ts=2026-09-22T13:47:02Z level=info event=scheduler_restarting addr=lb:500516297worker1 # [ 107.788820] nix-grpc-daemon[1317]: ts=2026-09-22T13:47:02Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:388486298worker1 # [ 107.793257] nix-grpc-daemon[1317]: ts=2026-09-22T13:47:02Z level=info event=worker_gone addr=192.168.1.3:500516299worker1 # [ 107.803805] nix-daemon[848]: reaped child process 1377, status = succeeded6300worker1 # [ 107.909385] nix-grpc-daemon[1317]: E0000 00:00:1790084822.181731 1362 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6301worker1 # [ 107.914291] nix-grpc-daemon[1317]: E0000 00:00:1790084822.182024 1362 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 10, started true, shutdown true6302worker2 # [ 107.979935] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:03Z level=info event=scheduler_connected addr=lb:500516303worker2 # [ 107.981790] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6304worker2 # [ 108.182457] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:03Z level=info event=scheduler_connected addr=lb:500516305worker2 # [ 108.184380] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6306worker2 # [ 108.384969] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:03Z level=info event=scheduler_connected addr=lb:500516307worker2 # [ 108.386847] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6308worker2 # [ 108.587534] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:03Z level=info event=scheduler_connected addr=lb:500516309worker2 # [ 108.589317] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6310worker2 # [ 108.791570] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:03Z level=info event=scheduler_connected addr=lb:500516311worker2 # [ 108.795168] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6312worker2 # [ 108.996196] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:04Z level=info event=scheduler_connected addr=lb:500516313worker2 # [ 109.000222] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6314worker2 # [ 109.200949] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:04Z level=info event=scheduler_connected addr=lb:500516315worker2 # [ 109.202967] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6316worker2 # [ 109.403745] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:04Z level=info event=scheduler_connected addr=lb:500516317worker2 # [ 109.405558] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6318worker2 # [ 109.606276] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:04Z level=info event=scheduler_connected addr=lb:500516319worker2 # [ 109.608269] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6320worker2 # [ 109.810788] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:04Z level=info event=scheduler_connected addr=lb:500516321worker2 # [ 109.813908] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6322worker1 # [ 109.988078] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6323worker1 # [ 109.991093] systemd[1]: Stopped Nix worker-protocol over gRPC.6324worker1 # [ 109.995240] systemd[1]: nix-grpc-daemon.service: Consumed 215ms CPU time over 11.492s wall clock time, 14.1M memory peak, 10.3K incoming IP traffic, 15.1K outgoing IP traffic.6325worker2 # [ 110.011841] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:05Z level=info event=scheduler_connected addr=lb:500516326worker1 # [ 110.006332] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.6327worker2 # [ 110.015868] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6328worker1 # [ 110.010501] systemd[1]: Closed Nix worker-protocol over gRPC.6329worker1 # [ 110.013710] systemd[1]: nix-grpc-daemon.socket: Consumed 527us CPU time over 11.529s wall clock time, 256K memory peak, 23K incoming IP traffic, 17.3K outgoing IP traffic.6330worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over6331worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.05 seconds)6332worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6333worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.02 seconds)6334worker2 # [ 110.216558] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:05Z level=info event=scheduler_connected addr=lb:500516335worker2 # [ 110.220310] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6336worker2 # [ 110.420880] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:05Z level=info event=scheduler_connected addr=lb:500516337worker2 # [ 110.422663] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6338worker2 # [ 110.623513] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:05Z level=info event=scheduler_connected addr=lb:500516339worker2 # [ 110.625335] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6340lb # [ 110.763214] niks3-server[1149]: 2026/09/22 13:47:05 INFO lead: acquired remote=192.168.1.4:404566341worker2 # [ 110.776901] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:05Z level=info event=scheduler_take_over6342worker2 # [ 110.827748] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:05Z level=info event=scheduler_connected addr=lb:500516343worker2 # [ 110.830980] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6344worker2 # [ 111.028969] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:06Z level=info event=scheduler_connected addr=lb:500516345worker2 # [ 111.033125] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6346worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6347worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6348worker2 # [ 111.233868] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:06Z level=info event=scheduler_connected addr=lb:500516349worker2 # [ 111.249152] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:06Z level=info event=oidc_jwks provider=mock keys=16350worker2 # [ 111.252661] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:06Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:368846351worker2 # [ 111.256488] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:06Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06352worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6353worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6354client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&26355worker2 # [ 112.348865] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:436406356client # don't know how to build these paths:6357client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6358worker2 # [ 112.356996] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436406359worker2 # [ 112.373607] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:07Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=06360worker2 # [ 112.378764] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:07Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.4:50051 clients=16361worker2 # [ 112.416526] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406362worker2 # [ 112.426564] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436406363client # copying 1 paths to 'lb:50051'...6364client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...6365worker2 # [ 112.436783] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436406366worker2 # [ 112.448333] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6367worker2 # [ 112.542752] nix-daemon[870]: accepted connection from pid 1483, user nix-grpc-daemon (trusted)6368worker2 # [ 112.558209] nix-daemon[870]: reaped child process 1486, status = succeeded6369worker2 # [ 112.630911] nix-daemon[870]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6370worker2 # [ 112.644409] nix-daemon[870]: reaped child process 1491, status = succeeded6371lb # [ 112.637910] niks3-server[1149]: 2026/09/22 13:47:07 INFO Received uploads request method=POST path=/api/pending_closures6372worker2 # [ 112.654182] nix-grpc-daemon[1336]: time=2026-09-22T13:47:07.716Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6373worker2 # [ 112.655541] nix-grpc-daemon[1336]: time=2026-09-22T13:47:07.716Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"6374lb # [ 112.649882] niks3-server[1149]: 2026/09/22 13:47:07 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst6375lb # [ 112.656533] niks3-server[1149]: 2026/09/22 13:47:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6376lb # [ 112.659168] niks3-server[1149]: 2026/09/22 13:47:07 INFO Signed narinfos id=43 count=16377worker2 # [ 112.670909] nix-grpc-daemon[1336]: time=2026-09-22T13:47:07.733Z level=INFO msg="Uploading 1 narinfos"6378lb # [ 112.664305] niks3-server[1149]: 2026/09/22 13:47:07 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls6379lb # [ 112.668968] niks3-server[1149]: 2026/09/22 13:47:07 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6380lb # [ 112.671478] niks3-server[1149]: 2026/09/22 13:47:07 INFO Completed upload id=436381worker2 # [ 112.684444] nix-grpc-daemon[1336]: time=2026-09-22T13:47:07.747Z level=INFO msg="Upload complete. (224ms)"6382lb # [ 112.674912] niks3-server[1149]: 2026/09/22 13:47:07 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo6383worker2 # [ 112.686622] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=0 paths=1 nar_bytes_in=5206384worker2 # [ 112.690924] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406385worker2 # [ 112.697191] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6386worker2 # [ 112.699519] nix-daemon[870]: reaped child process 1481, status = succeeded6387worker2 # [ 112.704461] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6388client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...6389client # farm-a> worker2: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6390client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6391client # farm-a> LOG-farm-a-on-standby6392worker2 # [ 113.939560] nix-daemon[870]: reaped child process 1495, status = succeeded6393worker2 # [ 114.026982] nix-daemon[870]: accepted connection from pid 1511, user nix-grpc-daemon (trusted)6394worker2 # [ 114.039657] nix-daemon[870]: reaped child process 1514, status = succeeded6395lb # [ 114.034841] niks3-server[1149]: 2026/09/22 13:47:09 INFO Received uploads request method=POST path=/api/pending_closures6396worker2 # [ 114.057163] nix-grpc-daemon[1336]: time=2026-09-22T13:47:09.119Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6397worker2 # [ 114.058664] nix-grpc-daemon[1336]: time=2026-09-22T13:47:09.121Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"6398lb # [ 114.080118] niks3-server[1149]: 2026/09/22 13:47:09 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst6399lb # [ 114.089883] niks3-server[1149]: 2026/09/22 13:47:09 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv6400lb # [ 114.096339] niks3-server[1149]: 2026/09/22 13:47:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6401lb # [ 114.103531] niks3-server[1149]: 2026/09/22 13:47:09 INFO Signed narinfos id=44 count=16402worker2 # [ 114.115809] nix-grpc-daemon[1336]: time=2026-09-22T13:47:09.178Z level=INFO msg="Uploading 1 narinfos"6403lb # [ 114.109150] niks3-server[1149]: 2026/09/22 13:47:09 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls6404lb # [ 114.120132] niks3-server[1149]: 2026/09/22 13:47:09 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6405lb # [ 114.124622] niks3-server[1149]: 2026/09/22 13:47:09 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo6406lb # [ 114.127673] niks3-server[1149]: 2026/09/22 13:47:09 INFO Completed upload id=446407worker2 # [ 114.139487] nix-grpc-daemon[1336]: time=2026-09-22T13:47:09.202Z level=INFO msg="Upload complete. (227ms)"6408worker2 # [ 114.144367] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=16409worker2 # [ 114.150276] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:09Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=06410worker2 # [ 114.152577] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:09Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.4:50051 clients=16411worker2 # [ 114.159846] nix-daemon[870]: reaped child process 1493, status = succeeded6412worker2 # [ 114.196174] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406413worker2 # [ 114.204563] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436406414worker2 # [ 114.214307] nix-grpc-daemon[1336]: time=2026-09-22T13:47:09.276Z level=INFO msg="All 1 paths already cached"6415client # copying 1 paths to 'lb:50051'...6416client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...6417worker2 # [ 114.220434] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436406418worker2 # [ 114.225577] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6419worker2 # [ 114.312486] nix-daemon[870]: accepted connection from pid 1520, user nix-grpc-daemon (trusted)6420worker2 # [ 114.325828] nix-daemon[870]: reaped child process 1523, status = succeeded6421worker2 # [ 114.408050] nix-daemon[870]: accepted connection from pid 1525, user nix-grpc-daemon (trusted)6422worker2 # [ 114.422250] nix-daemon[870]: reaped child process 1528, status = succeeded6423lb # [ 114.416429] niks3-server[1149]: 2026/09/22 13:47:09 INFO Received uploads request method=POST path=/api/pending_closures6424worker2 # [ 114.436543] nix-grpc-daemon[1336]: time=2026-09-22T13:47:09.498Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6425worker2 # [ 114.437907] nix-grpc-daemon[1336]: time=2026-09-22T13:47:09.499Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"6426lb # [ 114.440857] niks3-server[1149]: 2026/09/22 13:47:09 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst6427lb # [ 114.453243] niks3-server[1149]: 2026/09/22 13:47:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6428lb # [ 114.457929] niks3-server[1149]: 2026/09/22 13:47:09 INFO Signed narinfos id=45 count=16429worker2 # [ 114.473136] nix-grpc-daemon[1336]: time=2026-09-22T13:47:09.535Z level=INFO msg="Uploading 1 narinfos"6430lb # [ 114.468956] niks3-server[1149]: 2026/09/22 13:47:09 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls6431lb # [ 114.481317] niks3-server[1149]: 2026/09/22 13:47:09 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6432lb # [ 114.488238] niks3-server[1149]: 2026/09/22 13:47:09 INFO Completed upload id=456433worker2 # [ 114.502283] nix-grpc-daemon[1336]: time=2026-09-22T13:47:09.564Z level=INFO msg="Upload complete. (269ms)"6434lb # [ 114.492033] niks3-server[1149]: 2026/09/22 13:47:09 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo6435worker2 # [ 114.504536] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=0 paths=1 nar_bytes_in=6406436worker2 # [ 114.508349] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406437worker2 # [ 114.515414] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6438worker2 # [ 114.517845] nix-daemon[870]: reaped child process 1518, status = succeeded6439worker2 # [ 114.523155] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6440client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...6441client # farm-b> worker2: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6442client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6443lb # [ 115.586338] envoy[917]: conn peer=[2001:db8:1::1]:33892 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9735 tx=4448 ms=56446444lb # [ 115.590221] envoy[917]: conn peer=[2001:db8:1::3]:53942 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=950 tx=293 ms=93276445client # farm-b> LOG-farm-b-on-standby6446worker2 # [ 115.739324] nix-daemon[870]: reaped child process 1532, status = succeeded6447worker2 # [ 115.812595] nix-daemon[870]: accepted connection from pid 1548, user nix-grpc-daemon (trusted)6448worker2 # [ 115.826329] nix-daemon[870]: reaped child process 1551, status = succeeded6449lb # [ 115.820626] niks3-server[1149]: 2026/09/22 13:47:10 INFO Received uploads request method=POST path=/api/pending_closures6450worker2 # [ 115.841913] nix-grpc-daemon[1336]: time=2026-09-22T13:47:10.904Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6451worker2 # [ 115.843566] nix-grpc-daemon[1336]: time=2026-09-22T13:47:10.906Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"6452lb # [ 115.860861] niks3-server[1149]: 2026/09/22 13:47:10 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst6453lb # [ 115.876426] niks3-server[1149]: 2026/09/22 13:47:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6454lb # [ 115.882584] niks3-server[1149]: 2026/09/22 13:47:11 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv6455lb # [ 115.895788] niks3-server[1149]: 2026/09/22 13:47:11 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls6456worker2 # [ 115.917557] nix-grpc-daemon[1336]: time=2026-09-22T13:47:10.980Z level=INFO msg="Uploading 1 narinfos"6457lb # [ 115.905973] niks3-server[1149]: 2026/09/22 13:47:11 INFO Signed narinfos id=46 count=16458lb # [ 115.911894] niks3-server[1149]: 2026/09/22 13:47:11 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6459lb # [ 115.916456] niks3-server[1149]: 2026/09/22 13:47:11 INFO Completed upload id=466460worker2 # [ 115.929619] nix-grpc-daemon[1336]: time=2026-09-22T13:47:10.992Z level=INFO msg="Upload complete. (199ms)"6461worker2 # [ 115.933294] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=16462lb # [ 115.922760] niks3-server[1149]: 2026/09/22 13:47:11 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo6463worker2 # [ 115.939236] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:11Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=06464worker2 # [ 115.941240] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:11Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.4:50051 clients=16465worker2 # [ 115.946517] nix-daemon[870]: reaped child process 1530, status = succeeded6466worker2 # [ 115.982891] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406467worker2 # [ 115.991221] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436406468worker2 # [ 116.000835] nix-grpc-daemon[1336]: time=2026-09-22T13:47:11.063Z level=INFO msg="All 2 paths already cached"6469client # copying 1 paths to 'lb:50051'...6470worker2 # [ 116.006623] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436406471client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...6472worker2 # [ 116.011607] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6473worker2 # [ 116.091787] nix-daemon[870]: accepted connection from pid 1557, user nix-grpc-daemon (trusted)6474worker2 # [ 116.106479] nix-daemon[870]: reaped child process 1560, status = succeeded6475worker2 # [ 116.179677] nix-daemon[870]: accepted connection from pid 1562, user nix-grpc-daemon (trusted)6476lb # [ 116.183146] niks3-server[1149]: 2026/09/22 13:47:11 INFO Received uploads request method=POST path=/api/pending_closures6477worker2 # [ 116.195484] nix-daemon[870]: reaped child process 1565, status = succeeded6478worker2 # [ 116.200381] nix-grpc-daemon[1336]: time=2026-09-22T13:47:11.262Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6479worker2 # [ 116.201733] nix-grpc-daemon[1336]: time=2026-09-22T13:47:11.262Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"6480lb # [ 116.198994] niks3-server[1149]: 2026/09/22 13:47:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6481lb # [ 116.201037] niks3-server[1149]: 2026/09/22 13:47:11 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst6482lb # [ 116.203486] niks3-server[1149]: 2026/09/22 13:47:11 INFO Signed narinfos id=47 count=16483worker2 # [ 116.215152] nix-grpc-daemon[1336]: time=2026-09-22T13:47:11.277Z level=INFO msg="Uploading 1 narinfos"6484lb # [ 116.210540] niks3-server[1149]: 2026/09/22 13:47:11 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls6485lb # [ 116.211945] niks3-server[1149]: 2026/09/22 13:47:11 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6486lb # [ 116.217989] niks3-server[1149]: 2026/09/22 13:47:11 INFO Completed upload id=476487worker2 # [ 116.229801] nix-grpc-daemon[1336]: time=2026-09-22T13:47:11.292Z level=INFO msg="Upload complete. (211ms)"6488lb # [ 116.219658] niks3-server[1149]: 2026/09/22 13:47:11 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo6489worker2 # [ 116.231910] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=0 paths=1 nar_bytes_in=7686490worker2 # [ 116.235610] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406491worker2 # [ 116.241939] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6492worker2 # [ 116.244292] nix-daemon[870]: reaped child process 1555, status = succeeded6493worker2 # [ 116.249284] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6494client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...6495client # farm-top> worker2: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6496client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6497client # farm-top> LOG-farm-top-on-standby6498worker2 # [ 117.719144] nix-daemon[870]: reaped child process 1569, status = succeeded6499worker2 # [ 117.759393] nix-daemon[870]: accepted connection from pid 1585, user nix-grpc-daemon (trusted)6500worker2 # [ 117.774562] nix-daemon[870]: reaped child process 1588, status = succeeded6501lb # [ 117.767369] niks3-server[1149]: 2026/09/22 13:47:12 INFO Received uploads request method=POST path=/api/pending_closures6502worker2 # [ 117.787115] nix-grpc-daemon[1336]: time=2026-09-22T13:47:12.849Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6503worker2 # [ 117.788546] nix-grpc-daemon[1336]: time=2026-09-22T13:47:12.851Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"6504lb # [ 117.806168] niks3-server[1149]: 2026/09/22 13:47:12 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst6505lb # [ 117.815515] niks3-server[1149]: 2026/09/22 13:47:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6506lb # [ 117.820899] niks3-server[1149]: 2026/09/22 13:47:12 INFO Signed narinfos id=48 count=16507worker2 # [ 117.836179] nix-grpc-daemon[1336]: time=2026-09-22T13:47:12.898Z level=INFO msg="Uploading 1 narinfos"6508lb # [ 117.830494] niks3-server[1149]: 2026/09/22 13:47:12 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv6509lb # [ 117.839940] niks3-server[1149]: 2026/09/22 13:47:12 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6510lb # [ 117.847667] niks3-server[1149]: 2026/09/22 13:47:12 INFO Completed upload id=486511lb # [ 117.849880] niks3-server[1149]: 2026/09/22 13:47:12 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls6512worker2 # [ 117.859380] nix-grpc-daemon[1336]: time=2026-09-22T13:47:12.921Z level=INFO msg="Upload complete. (201ms)"6513lb # [ 117.851972] niks3-server[1149]: 2026/09/22 13:47:12 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo6514worker2 # [ 117.865761] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=16515worker2 # [ 117.882334] nix-daemon[870]: reaped child process 1567, status = succeeded6516client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby >&2, in 5.68 seconds)6517client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath6518client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag on-standby outPath, in 0.10 seconds)6519worker1 # [ 118.023456] systemd[1]: Listening on Nix worker-protocol over gRPC.6520worker1 # [ 118.031600] systemd[1]: Starting Nix worker-protocol over gRPC...6521worker1 # [ 118.120807] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:12Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6522worker1 # [ 118.122747] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:12Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6523worker1 # [ 118.125804] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:12Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16524worker1 # [ 118.130054] nix-grpc-daemon[1407]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6525worker1 # [ 118.132170] nix-grpc-daemon[1407]: E0000 00:00:1790084832.402572 1407 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6526worker1 # [ 118.138067] nix-grpc-daemon[1407]: E0000 00:00:1790084832.407938 1407 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6527worker1 # [ 118.141535] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:12Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6528worker1 # [ 118.144299] systemd[1]: Started Nix worker-protocol over gRPC.6529worker1 # [ 118.147134] nix-grpc-daemon[1407]: E0000 00:00:1790084832.418858 1448 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6530worker1 # [ 118.159407] nix-grpc-daemon[1407]: E0000 00:00:1790084832.432128 1448 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6531worker1 # [ 118.163062] nix-grpc-daemon[1407]: E0000 00:00:1790084832.434529 1448 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6532lb: must succeed: curl -sf localhost:9901/clusters6533worker1 # [ 118.173722] nix-grpc-daemon[1407]: E0000 00:00:1790084832.446444 1448 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6534worker1 # [ 118.181135] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:12Z level=info event=scheduler_connected addr=lb:500516535worker2 # [ 118.191134] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:13Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:368846536lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6537worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6538worker2 # [ 118.195416] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:13Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06539worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6540lb: must succeed: curl -sf localhost:9901/clusters6541lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6542lb: must succeed: curl -sf localhost:9901/clusters6543lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6544worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6545worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6546(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.33 seconds)6547subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6548client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs6549client # [ 118.379124] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs.6550client # Running as unit: rs.service; invocation ID: 5d2349b93801462aad52f80ada422d1b6551client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag rs, in 0.06 seconds)6552worker2 # [ 118.498238] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:436406553client # [ 118.520992] nix[1368]: don't know how to build these paths:6554client # [ 118.521954] nix[1368]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6555worker2 # [ 118.511617] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:13Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06556worker2 # [ 118.516724] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:13Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.4:50051 clients=16557worker2 # [ 118.554742] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406558worker2 # [ 118.565460] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436406559client # [ 118.590093] nix[1368]: copying 1 paths to 'lb:50051'...6560client # [ 118.591058] nix[1368]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6561worker2 # [ 118.574350] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436406562worker2 # [ 118.584163] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6563worker2 # [ 118.675175] nix-daemon[870]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6564worker2 # [ 118.689741] nix-daemon[870]: reaped child process 1604, status = succeeded6565worker2 # [ 118.763413] nix-daemon[870]: accepted connection from pid 1606, user nix-grpc-daemon (trusted)6566worker2 # [ 118.778166] nix-daemon[870]: reaped child process 1609, status = succeeded6567lb # [ 118.770609] niks3-server[1149]: 2026/09/22 13:47:13 INFO Received uploads request method=POST path=/api/pending_closures6568worker2 # [ 118.791326] nix-grpc-daemon[1336]: time=2026-09-22T13:47:13.853Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6569worker2 # [ 118.792646] nix-grpc-daemon[1336]: time=2026-09-22T13:47:13.853Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6570lb # [ 118.799085] niks3-server[1149]: 2026/09/22 13:47:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6571lb # [ 118.807441] niks3-server[1149]: 2026/09/22 13:47:13 INFO Signed narinfos id=49 count=16572worker2 # [ 118.822728] nix-grpc-daemon[1336]: time=2026-09-22T13:47:13.885Z level=INFO msg="Uploading 1 narinfos"6573lb # [ 118.812271] niks3-server[1149]: 2026/09/22 13:47:13 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6574lb # [ 118.819503] niks3-server[1149]: 2026/09/22 13:47:13 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6575lb # [ 118.829690] niks3-server[1149]: 2026/09/22 13:47:13 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6576lb # [ 118.836588] niks3-server[1149]: 2026/09/22 13:47:13 INFO Completed upload id=496577worker2 # [ 118.848192] nix-grpc-daemon[1336]: time=2026-09-22T13:47:13.910Z level=INFO msg="Upload complete. (252ms)"6578lb # [ 118.839573] niks3-server[1149]: 2026/09/22 13:47:13 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6579worker2 # [ 118.850456] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=0 paths=1 nar_bytes_in=4166580worker2 # [ 118.854274] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436406581worker2 # [ 118.862855] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6582worker2 # [ 118.869324] nix-daemon[870]: reaped child process 1599, status = succeeded6583client # [ 118.887972] nix[1368]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6584worker2 # [ 118.872414] nix-daemon[870]: accepted connection from pid 1276, user nix-grpc-daemon (trusted)6585lb: must succeed: curl -sf localhost:9901/clusters6586lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6587lb: must succeed: curl -sf localhost:9901/clusters6588lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6589lb: must succeed: curl -sf localhost:9901/clusters6590lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6591lb: must succeed: curl -sf localhost:9901/clusters6592lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)6593worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6594worker1 # [ 120.961269] systemd-journald[285]: Received client request to rotate journal, rotating.6595worker1 # [ 120.978790] systemd-journald[285]: Vacuuming done, freed 0B of archived journals from /var/log/journal/7c338c9042ae44478d0384c2b31d22ed.6596worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.06 seconds)6597worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service6598worker2 # [ 120.903764] systemd[1]: Stopping Nix worker-protocol over gRPC...6599worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)6600client: must succeed: sleep 5.06601worker2 # [ 121.171079] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:16Z level=info event=draining6602lb # [ 125.586822] envoy[917]: conn peer=[2001:db8:1::1]:48298 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9612 tx=4369 ms=55296603client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6604worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6605worker2 # [ 126.346795] nix-daemon[870]: reaped child process 1613, status = succeeded6606worker2 # [ 126.389064] nix-daemon[870]: accepted connection from pid 1654, user nix-grpc-daemon (trusted)6607worker2 # [ 126.403095] nix-daemon[870]: reaped child process 1657, status = succeeded6608lb # [ 126.398423] niks3-server[1149]: 2026/09/22 13:47:21 INFO Received uploads request method=POST path=/api/pending_closures6609worker2 # [ 126.418689] nix-grpc-daemon[1336]: time=2026-09-22T13:47:21.481Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6610worker2 # [ 126.420206] nix-grpc-daemon[1336]: time=2026-09-22T13:47:21.482Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6611lb # [ 126.436444] niks3-server[1149]: 2026/09/22 13:47:21 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6612lb # [ 126.449768] niks3-server[1149]: 2026/09/22 13:47:21 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6613lb # [ 126.457943] niks3-server[1149]: 2026/09/22 13:47:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6614lb # [ 126.463137] niks3-server[1149]: 2026/09/22 13:47:21 INFO Signed narinfos id=50 count=16615worker2 # [ 126.477782] nix-grpc-daemon[1336]: time=2026-09-22T13:47:21.540Z level=INFO msg="Uploading 1 narinfos"6616lb # [ 126.472565] niks3-server[1149]: 2026/09/22 13:47:21 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6617lb # [ 126.479781] niks3-server[1149]: 2026/09/22 13:47:21 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6618lb # [ 126.486578] niks3-server[1149]: 2026/09/22 13:47:21 INFO Completed upload id=506619lb # [ 126.488951] niks3-server[1149]: 2026/09/22 13:47:21 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6620worker2 # [ 126.499868] nix-grpc-daemon[1336]: time=2026-09-22T13:47:21.562Z level=INFO msg="Upload complete. (213ms)"6621worker2 # [ 126.508427] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43640 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16622worker2 # [ 126.522960] nix-daemon[870]: reaped child process 1611, status = succeeded6623client # [ 126.553936] systemd[1]: rs.service: Deactivated successfully.6624client # [ 126.557887] systemd[1]: rs.service: Consumed 126ms CPU time over 8.178s wall clock time, 43.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.6625worker2 # [ 127.172363] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:22Z level=info event=signal_exit6626lb # [ 127.768963] niks3-server[1149]: 2026/09/22 13:47:22 INFO lead: released remote=192.168.1.4:404566627lb # [ 128.142956] niks3-server[1149]: 2026/09/22 13:47:23 INFO lead: acquired remote=192.168.1.3:364926628worker1 # [ 128.149929] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:22Z level=info event=scheduler_take_over6629worker1 # [ 128.772647] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:23Z level=info event=scheduler_restarting addr=lb:500516630worker2 # [ 128.785651] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:23Z level=info event=scheduler_restarting6631worker2 # [ 128.790103] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:23Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:368846632worker2 # [ 128.794383] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:23Z level=info event=worker_gone addr=192.168.1.3:500516633worker2 # [ 128.797443] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:23Z level=info event=scheduler_restarting addr=lb:500516634worker2 # [ 128.800642] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:23Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:368846635worker2 # [ 128.804214] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:23Z level=info event=worker_gone addr=192.168.1.4:500516636worker2 # [ 128.824424] nix-daemon[870]: reaped child process 1328, status = succeeded6637worker1 # [ 128.973182] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:23Z level=info event=scheduler_connected addr=lb:500516638worker1 # [ 128.979978] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:23Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:539546639worker1 # [ 128.984484] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:23Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06640worker2 # [ 128.998624] nix-grpc-daemon[1276]: ts=2026-09-22T13:47:24Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6641worker2 # [ 129.234341] nix-grpc-daemon[1276]: E0000 00:00:1790084844.296703 1321 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true6642worker2 # [ 129.239286] nix-grpc-daemon[1276]: E0000 00:00:1790084844.296977 1321 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6643worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.17 seconds)6644client: waiting for success: ! systemctl is-active rs6645client: (finished: waiting for success: ! systemctl is-active rs, in 0.04 seconds)6646client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6647client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.03 seconds)6648client: must fail: journalctl -u rs | grep -q 'reconnecting'6649client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.01 seconds)6650lb: must succeed: curl -sf localhost:9901/clusters6651lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6652worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6653worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6654worker2 # [ 130.281765] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6655worker2 # [ 130.284708] systemd[1]: Stopped Nix worker-protocol over gRPC.6656worker2 # [ 130.287755] systemd[1]: nix-grpc-daemon.service: Consumed 2.112s CPU time over 36.214s wall clock time, 46.8M memory peak, 85.9K incoming IP traffic, 217.6K outgoing IP traffic.6657worker2 # [ 130.302899] systemd[1]: Starting scheduler-token.service...6658worker2 # [ 130.345182] systemd[1]: scheduler-token.service: Deactivated successfully.6659worker2 # [ 130.347187] systemd[1]: Finished scheduler-token.service.6660worker2 # [ 130.347904] systemd[1]: scheduler-token.service: Consumed 25ms CPU time over 44ms wall clock time, 2.3M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6661worker2 # [ 130.352156] systemd[1]: Starting Nix worker-protocol over gRPC...6662worker2 # [ 130.429394] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:25Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6663worker2 # [ 130.431506] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:25Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6664worker2 # [ 130.434693] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:25Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16665worker2 # [ 130.439093] nix-grpc-daemon[1667]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6666worker2 # [ 130.440410] nix-grpc-daemon[1667]: E0000 00:00:1790084845.501311 1667 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6667worker2 # [ 130.446174] nix-grpc-daemon[1667]: E0000 00:00:1790084845.506693 1667 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6668worker2 # [ 130.449436] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:25Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6669worker2 # [ 130.451366] systemd[1]: Started Nix worker-protocol over gRPC.6670worker2 # [ 130.455067] nix-grpc-daemon[1667]: E0000 00:00:1790084845.517103 1707 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6671lb: must succeed: curl -sf localhost:9901/clusters6672worker2 # [ 130.466491] nix-grpc-daemon[1667]: E0000 00:00:1790084845.529156 1707 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6673worker2 # [ 130.471355] nix-grpc-daemon[1667]: E0000 00:00:1790084845.531538 1707 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6674worker2 # [ 130.481930] nix-grpc-daemon[1667]: E0000 00:00:1790084845.544554 1707 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6675worker2 # [ 130.489790] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:25Z level=info event=scheduler_connected addr=lb:500516676lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6677worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6678worker1 # [ 130.489452] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:24Z level=info event=oidc_jwks provider=mock keys=16679worker1 # [ 130.493491] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:24Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:539546680worker1 # [ 130.497949] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:24Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06681worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6682(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.23 seconds)6683subtest: scheduler restart mid-build: build finishes, no double build6684client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid6685client # [ 130.605724] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid.6686client # Running as unit: mid.service; invocation ID: 8c3138a4453a480d91546c8a44e83f0e6687client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag mid, in 0.06 seconds)6688worker1 # [ 130.722620] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:539646689worker1 # [ 130.732229] nix-daemon[848]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6690client # [ 130.767412] nix[1403]: don't know how to build these paths:6691client # [ 130.768263] nix[1403]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6692worker1 # [ 130.751224] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:25Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06693worker1 # [ 130.756066] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:25Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16694worker1 # [ 130.799526] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:539646695worker1 # [ 130.810296] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 12 ms (attempt 1/5)6696worker1 # [ 130.830391] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 193 ms (attempt 2/5)6697worker1 # [ 131.031082] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 394 ms (attempt 3/5)6698worker1 # [ 131.427182] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 425 ms (attempt 4/5)6699worker1 # [ 131.854875] nix-grpc-daemon[1407]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6700worker1 # [ 131.858414] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6701worker1 # [ 131.876072] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:539646702client # [ 131.910990] nix[1403]: copying 1 paths to 'lb:50051'...6703worker1 # [ 131.888233] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:539646704client # [ 131.915148] nix[1403]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6705worker1 # [ 131.901196] nix-daemon[848]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6706worker1 # [ 131.919807] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:26Z level=info event=niks3_push_started pid=15396707worker1 # [ 132.022632] nix-grpc-daemon[1539]: time=2026-09-22T13:47:26.294Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6708worker1 # [ 132.108136] nix-daemon[848]: accepted connection from pid 1552, user nix-grpc-daemon (trusted)6709worker1 # [ 132.118640] nix-daemon[848]: reaped child process 1555, status = succeeded6710worker1 # [ 132.204635] nix-daemon[848]: accepted connection from pid 1557, user nix-grpc-daemon (trusted)6711worker1 # [ 132.216441] nix-daemon[848]: reaped child process 1560, status = succeeded6712lb # [ 132.222367] niks3-server[1149]: 2026/09/22 13:47:27 INFO Received uploads request method=POST path=/api/pending_closures6713worker1 # [ 132.233327] nix-grpc-daemon[1539]: time=2026-09-22T13:47:26.506Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6714worker1 # [ 132.236362] nix-grpc-daemon[1539]: time=2026-09-22T13:47:26.507Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6715lb # [ 132.259107] niks3-server[1149]: 2026/09/22 13:47:27 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6716lb # [ 132.268268] niks3-server[1149]: 2026/09/22 13:47:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6717worker1 # [ 132.277921] nix-grpc-daemon[1539]: time=2026-09-22T13:47:26.550Z level=INFO msg="Uploading 1 narinfos"6718lb # [ 132.274881] niks3-server[1149]: 2026/09/22 13:47:27 INFO Signed narinfos id=51 count=16719lb # [ 132.288242] niks3-server[1149]: 2026/09/22 13:47:27 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6720lb # [ 132.296807] niks3-server[1149]: 2026/09/22 13:47:27 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6721worker1 # [ 132.304916] nix-grpc-daemon[1539]: time=2026-09-22T13:47:26.577Z level=INFO msg="Upload complete. (279ms)"6722lb # [ 132.303043] niks3-server[1149]: 2026/09/22 13:47:27 INFO Completed upload id=516723lb # [ 132.307673] niks3-server[1149]: 2026/09/22 13:47:27 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo6724worker1 # [ 132.308122] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53964 duration_s=0 paths=1 nar_bytes_in=4246725worker1 # [ 132.313464] nix-grpc-daemon[1407]: ts=2026-09-22T13:47:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:539646726worker1 # [ 132.321611] nix-daemon[848]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6727worker1 # [ 132.326038] nix-daemon[848]: reaped child process 1537, status = succeeded6728worker1 # [ 132.330688] nix-daemon[848]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6729client # [ 132.355461] nix[1403]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6730lb: must succeed: curl -sf localhost:9901/clusters6731lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6732lb: must succeed: curl -sf localhost:9901/clusters6733lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6734worker1: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6735worker1 # [ 134.077531] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1407 (nix-grpc-daemon) on client request.6736worker1 # [ 134.081440] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6737lb # [ 134.087093] niks3-server[1149]: 2026/09/22 13:47:29 INFO lead: released remote=192.168.1.3:364926738worker2 # [ 134.103725] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:29Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6739client # [ 134.120580] nix[1403]: n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6740worker1 # [ 134.099163] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6741worker1 # [ 134.101297] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6742client # [ 134.125370] nix[1403]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6743worker1 # [ 134.106324] systemd[1]: nix-grpc-daemon.service: Consumed 413ms CPU time over 16.066s wall clock time, 35.5M memory peak, 14.5K incoming IP traffic, 26.1K outgoing IP traffic.6744worker1 # [ 134.108536] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6745worker1 # [ 134.114552] systemd[1]: Starting Nix worker-protocol over gRPC...6746worker1 # [ 134.140208] nix-daemon[848]: reaped child process 1562, status = succeeded6747worker1 # [ 134.142406] nix-daemon[848]: reaped child process 1525, status = succeeded6748worker1 # [ 134.152957] nix-daemon[1564]: unexpected Nix daemon error: error: interrupted by the user6749worker1 # [ 134.161712] nix-daemon[848]: reaped child process 1564, status = failed with exit code 16750worker1 # [ 134.203805] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6751worker1 # [ 134.205774] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6752worker1 # [ 134.208821] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16753worker1 # [ 134.213055] nix-grpc-daemon[1600]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6754worker1 # [ 134.215167] nix-grpc-daemon[1600]: E0000 00:00:1790084848.485533 1600 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6755worker1 # [ 134.221063] nix-grpc-daemon[1600]: E0000 00:00:1790084848.490895 1600 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6756worker1 # [ 134.224093] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6757worker1 # [ 134.227589] systemd[1]: Started Nix worker-protocol over gRPC.6758lb # [ 134.225887] niks3-server[1149]: 2026/09/22 13:47:29 INFO lead: acquired remote=192.168.1.3:549886759worker1 # [ 134.232123] nix-grpc-daemon[1600]: E0000 00:00:1790084848.501957 1642 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6760worker1 # [ 134.235264] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=info event=scheduler_take_over6761worker1: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.19 seconds)6762client: must succeed: sleep 5.06763worker1 # [ 134.243843] nix-grpc-daemon[1600]: E0000 00:00:1790084848.516490 1642 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6764worker1 # [ 134.247130] nix-grpc-daemon[1600]: E0000 00:00:1790084848.518682 1642 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6765worker1 # [ 134.257712] nix-grpc-daemon[1600]: E0000 00:00:1790084848.530389 1642 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6766worker1 # [ 134.266904] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=info event=scheduler_connected addr=lb:500516767worker1 # [ 134.273323] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:484366768worker1 # [ 134.274951] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06769worker2 # [ 134.303959] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:29Z level=info event=scheduler_connected addr=lb:500516770worker1 # [ 134.299626] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=info event=oidc_jwks provider=mock keys=16771worker1 # [ 134.301060] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:484366772worker1 # [ 134.302696] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06773worker1 # [ 134.609619] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06774worker1 # [ 134.614966] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16775worker1 # [ 134.661162] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:484526776worker1 # [ 134.671173] nix-daemon[848]: accepted connection from pid 1600, user nix-grpc-daemon (trusted)6777worker1 # [ 134.685901] nix-daemon[848]: accepted connection from pid 1600, user nix-grpc-daemon (trusted)6778client # [ 134.724490] nix[1403]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6779worker1 # [ 134.701194] nix-daemon[848]: accepted connection from pid 1600, user nix-grpc-daemon (trusted)6780lb # [ 135.586672] envoy[917]: conn peer=[2001:db8:1::1]:42152 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2960 tx=1516 ms=80186781lb # [ 135.588445] envoy[917]: conn peer=[2001:db8:1::4]:48192 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=35600 tx=2164 ms=347806782lb # [ 135.590088] envoy[917]: conn peer=[2001:db8:1::3]:40802 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=794 tx=311 ms=159166783client: (finished: must succeed: sleep 5.0, in 5.03 seconds)6784client: waiting for success: ! systemctl is-active mid6785worker1 # [ 139.527300] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:33Z level=info event=niks3_push_started pid=16846786worker1 # [ 139.560292] nix-daemon[848]: reaped child process 1659, status = succeeded6787worker1 # [ 139.633907] nix-grpc-daemon[1684]: time=2026-09-22T13:47:33.906Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""6788worker1 # [ 139.716891] nix-daemon[848]: accepted connection from pid 1697, user nix-grpc-daemon (trusted)6789worker1 # [ 139.728130] nix-daemon[848]: reaped child process 1700, status = succeeded6790lb # [ 139.732812] niks3-server[1149]: 2026/09/22 13:47:34 INFO Received uploads request method=POST path=/api/pending_closures6791worker1 # [ 139.739846] nix-grpc-daemon[1684]: time=2026-09-22T13:47:34.012Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6792worker1 # [ 139.741399] nix-grpc-daemon[1684]: time=2026-09-22T13:47:34.014Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6793lb # [ 139.759116] niks3-server[1149]: 2026/09/22 13:47:34 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6794lb # [ 139.764365] niks3-server[1149]: 2026/09/22 13:47:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6795lb # [ 139.766353] niks3-server[1149]: 2026/09/22 13:47:34 INFO Signed narinfos id=52 count=16796worker1 # [ 139.769230] nix-grpc-daemon[1684]: time=2026-09-22T13:47:34.041Z level=INFO msg="Uploading 1 narinfos"6797lb # [ 139.776759] niks3-server[1149]: 2026/09/22 13:47:34 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6798lb # [ 139.779371] niks3-server[1149]: 2026/09/22 13:47:34 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6799lb # [ 139.783036] niks3-server[1149]: 2026/09/22 13:47:34 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6800lb # [ 139.786382] niks3-server[1149]: 2026/09/22 13:47:34 INFO Completed upload id=526801worker1 # [ 139.788827] nix-grpc-daemon[1684]: time=2026-09-22T13:47:34.061Z level=INFO msg="Upload complete. (151ms)"6802worker1 # [ 139.791333] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48452 duration_s=5 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16803lb # [ 139.790810] niks3-server[1149]: 2026/09/22 13:47:34 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6804worker1 # [ 139.799885] nix-daemon[848]: reaped child process 1657, status = succeeded6805client # [ 139.849852] systemd[1]: mid.service: Deactivated successfully.6806client # [ 139.853558] systemd[1]: mid.service: Consumed 146ms CPU time over 9.247s wall clock time, 43.4M memory peak, 8.7K incoming IP traffic, 10.4K outgoing IP traffic.6807client: (finished: waiting for success: ! systemctl is-active mid, in 1.09 seconds)6808client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6809client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.03 seconds)6810(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.03 seconds)6811subtest: interrupting the client stops the build on the worker6812client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr6813client # [ 140.611965] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr.6814client # Running as unit: intr.service; invocation ID: 69ad2647762c4184965a80097256c2016815client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag intr, in 0.03 seconds)6816worker2 # [ 140.725339] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:531666817worker2 # [ 140.734678] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)6818client # [ 140.761266] nix[1439]: don't know how to build these paths:6819client # [ 140.762137] nix[1439]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6820worker1 # [ 140.744494] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:35Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06821worker1 # [ 140.748511] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:35Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=16822worker2 # [ 140.796788] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531666823worker2 # [ 140.806176] nix-grpc-daemon[1667]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 36 ms (attempt 1/5)6824worker2 # [ 140.848216] nix-grpc-daemon[1667]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 35 ms (attempt 2/5)6825worker2 # [ 140.889709] nix-grpc-daemon[1667]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 103 ms (attempt 3/5)6826worker2 # [ 141.000291] nix-grpc-daemon[1667]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 394 ms (attempt 4/5)6827worker2 # [ 141.396764] nix-grpc-daemon[1667]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6828worker2 # [ 141.400656] nix-grpc-daemon[1667]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6829worker2 # [ 141.417662] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:531666830client # [ 141.444304] nix[1439]: copying 1 paths to 'lb:50051'...6831client # [ 141.448146] nix[1439]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6832worker2 # [ 141.432464] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:531666833worker2 # [ 141.440830] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)6834worker2 # [ 141.463235] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:36Z level=info event=niks3_push_started pid=17596835worker2 # [ 141.563319] nix-grpc-daemon[1759]: time=2026-09-22T13:47:36.625Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker2.key ca=""6836worker2 # [ 141.649051] nix-daemon[870]: accepted connection from pid 1771, user nix-grpc-daemon (trusted)6837worker2 # [ 141.661820] nix-daemon[870]: reaped child process 1774, status = succeeded6838worker2 # [ 141.745203] nix-daemon[870]: accepted connection from pid 1776, user nix-grpc-daemon (trusted)6839worker2 # [ 141.767923] nix-daemon[870]: reaped child process 1782, status = succeeded6840lb # [ 141.758702] niks3-server[1149]: 2026/09/22 13:47:36 INFO Received uploads request method=POST path=/api/pending_closures6841worker2 # [ 141.778482] nix-grpc-daemon[1759]: time=2026-09-22T13:47:36.841Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6842worker2 # [ 141.780248] nix-grpc-daemon[1759]: time=2026-09-22T13:47:36.842Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6843lb # [ 141.792653] niks3-server[1149]: 2026/09/22 13:47:36 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6844lb # [ 141.806223] niks3-server[1149]: 2026/09/22 13:47:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6845lb # [ 141.814237] niks3-server[1149]: 2026/09/22 13:47:36 INFO Signed narinfos id=53 count=16846worker2 # [ 141.829254] nix-grpc-daemon[1759]: time=2026-09-22T13:47:36.891Z level=INFO msg="Uploading 1 narinfos"6847lb # [ 141.824913] niks3-server[1149]: 2026/09/22 13:47:36 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6848lb # [ 141.834312] niks3-server[1149]: 2026/09/22 13:47:36 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6849lb # [ 141.841351] niks3-server[1149]: 2026/09/22 13:47:36 INFO Completed upload id=536850worker2 # [ 141.853220] nix-grpc-daemon[1759]: time=2026-09-22T13:47:36.915Z level=INFO msg="Upload complete. (286ms)"6851lb # [ 141.844344] niks3-server[1149]: 2026/09/22 13:47:36 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6852worker2 # [ 141.855831] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53166 duration_s=0 paths=1 nar_bytes_in=4246853worker2 # [ 141.858757] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531666854worker2 # [ 141.866288] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)6855worker2 # [ 141.871138] nix-daemon[870]: reaped child process 1757, status = succeeded6856client # [ 141.889858] nix[1439]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6857worker2 # [ 141.874115] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)6858client: must succeed: systemctl kill -s INT intr6859client # [ 142.912935] systemd[1]: intr.service: Sent signal SIGINT to main process 1439 (nix) on client request.6860client # [ 142.915343] nix[1439]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6861client: (finished: must succeed: systemctl kill -s INT intr, in 0.02 seconds)6862client # [ 142.918438] nix[1439]: error: interrupted by the user6863worker2 # [ 142.902688] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:37Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6864client # [ 142.936726] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6865client # [ 142.938455] systemd[1]: intr.service: Failed with result 'exit-code'.6866client # [ 142.940366] systemd[1]: intr.service: Consumed 114ms CPU time over 2.326s wall clock time, 43.4M memory peak, 6.5K incoming IP traffic, 8.3K outgoing IP traffic.6867worker2 # [ 143.092914] nix-daemon[870]: reaped child process 1785, status = succeeded6868worker2 # [ 143.094526] nix-daemon[1787]: unexpected Nix daemon error: error: interrupted by the user6869worker2 # [ 143.099492] nix-daemon[870]: reaped child process 1787, status = failed with exit code 16870(finished: subtest: interrupting the client stops the build on the worker, in 3.48 seconds)6871subtest: a deploy mid-build drains: the build finishes, the new generation takes over6872client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw6873client # [ 144.121705] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key" --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw.6874client # Running as unit: sw.service; invocation ID: 59f49c95be7e447ab0d6c1acbe4970036875client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/xm9d525p8sv28c6nivsdjrjpb8kgp3fa-slow.nix --argstr tag sw, in 0.06 seconds)6876worker1 # [ 144.232389] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:38Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:484526877client # [ 144.263878] nix[1459]: don't know how to build these paths:6878client # [ 144.264809] nix[1459]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6879worker1 # [ 144.246115] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:38Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06880worker1 # [ 144.250929] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:38Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=16881worker1 # [ 144.290323] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:484526882worker1 # [ 144.300865] nix-grpc-daemon[1600]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 48 ms (attempt 1/5)6883worker1 # [ 144.356499] nix-grpc-daemon[1600]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 186 ms (attempt 2/5)6884worker1 # [ 144.544583] nix-grpc-daemon[1600]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 270 ms (attempt 3/5)6885worker1 # [ 144.815636] nix-grpc-daemon[1600]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 548 ms (attempt 4/5)6886worker1 # [ 145.365552] nix-grpc-daemon[1600]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6887worker1 # [ 145.367139] nix-grpc-daemon[1600]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6888worker1 # [ 145.378690] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:484526889worker1 # [ 145.386398] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:484526890client # [ 145.411841] nix[1459]: copying 1 paths to 'lb:50051'...6891client # [ 145.413754] nix[1459]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6892worker1 # [ 145.396510] nix-daemon[848]: accepted connection from pid 1600, user nix-grpc-daemon (trusted)6893worker1 # [ 145.477964] nix-daemon[848]: accepted connection from pid 1740, user nix-grpc-daemon (trusted)6894worker1 # [ 145.490422] nix-daemon[848]: reaped child process 1743, status = succeeded6895worker1 # [ 145.570339] nix-daemon[848]: accepted connection from pid 1745, user nix-grpc-daemon (trusted)6896worker1 # [ 145.582363] nix-daemon[848]: reaped child process 1748, status = succeeded6897lb # [ 145.587783] niks3-server[1149]: 2026/09/22 13:47:40 INFO Received uploads request method=POST path=/api/pending_closures6898lb # [ 145.594776] envoy[917]: conn peer=[2001:db8:1::1]:32794 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3921 tx=1963 ms=90856899lb # [ 145.598529] envoy[917]: conn peer=[2001:db8:1::1]:50782 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2980 tx=830 ms=21926900worker1 # [ 145.605881] nix-grpc-daemon[1684]: time=2026-09-22T13:47:39.878Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6901worker1 # [ 145.607419] nix-grpc-daemon[1684]: time=2026-09-22T13:47:39.880Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6902lb # [ 145.622719] niks3-server[1149]: 2026/09/22 13:47:40 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6903lb # [ 145.638522] niks3-server[1149]: 2026/09/22 13:47:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6904lb # [ 145.642481] niks3-server[1149]: 2026/09/22 13:47:40 INFO Signed narinfos id=54 count=16905worker1 # [ 145.647877] nix-grpc-daemon[1684]: time=2026-09-22T13:47:39.920Z level=INFO msg="Uploading 1 narinfos"6906lb # [ 145.648298] niks3-server[1149]: 2026/09/22 13:47:40 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6907lb # [ 145.658746] niks3-server[1149]: 2026/09/22 13:47:40 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6908lb # [ 145.662061] niks3-server[1149]: 2026/09/22 13:47:40 INFO Completed upload id=546909worker1 # [ 145.664975] nix-grpc-daemon[1684]: time=2026-09-22T13:47:39.937Z level=INFO msg="Upload complete. (264ms)"6910worker1 # [ 145.667240] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48452 duration_s=0 paths=1 nar_bytes_in=4166911lb # [ 145.665161] niks3-server[1149]: 2026/09/22 13:47:40 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6912worker1 # [ 145.670468] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:484526913worker1 # [ 145.679750] nix-daemon[848]: accepted connection from pid 1600, user nix-grpc-daemon (trusted)6914worker1 # [ 145.682570] nix-daemon[848]: reaped child process 1738, status = succeeded6915worker1 # [ 145.686968] nix-daemon[848]: accepted connection from pid 1600, user nix-grpc-daemon (trusted)6916client # [ 145.711792] nix[1459]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6917worker1: must succeed: systemctl reload nix-grpc-daemon.service6918worker1 # [ 147.581377] systemd[1]: Reloading Nix worker-protocol over gRPC...6919worker1 # [ 147.609782] systemd[1]: Reloaded Nix worker-protocol over gRPC.6920worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.06 seconds)6921worker1: must succeed: pgrep -f 'read -t [9]0 x'6922worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.03 seconds)6923client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&26924worker2 # [ 147.805605] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:42Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:531666925client # don't know how to build these paths:6926client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6927worker2 # [ 147.814082] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:531666928worker1 # [ 147.827437] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:42Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06929worker1 # [ 147.830192] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:42Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16930worker2 # [ 147.880169] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531666931worker2 # [ 147.890897] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:531666932client # copying 1 paths to 'lb:50051'...6933worker2 # [ 147.901956] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:531666934client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6935worker2 # [ 147.929281] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)6936worker2 # [ 148.029917] nix-daemon[870]: accepted connection from pid 1838, user nix-grpc-daemon (trusted)6937worker2 # [ 148.043434] nix-daemon[870]: reaped child process 1841, status = succeeded6938worker2 # [ 148.125205] nix-daemon[870]: accepted connection from pid 1844, user nix-grpc-daemon (trusted)6939worker2 # [ 148.140095] nix-daemon[870]: reaped child process 1847, status = succeeded6940lb # [ 148.132823] niks3-server[1149]: 2026/09/22 13:47:43 INFO Received uploads request method=POST path=/api/pending_closures6941worker2 # [ 148.149111] nix-grpc-daemon[1759]: time=2026-09-22T13:47:43.211Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6942worker2 # [ 148.150591] nix-grpc-daemon[1759]: time=2026-09-22T13:47:43.213Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6943lb # [ 148.152062] niks3-server[1149]: 2026/09/22 13:47:43 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6944lb # [ 148.158266] niks3-server[1149]: 2026/09/22 13:47:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6945lb # [ 148.160546] niks3-server[1149]: 2026/09/22 13:47:43 INFO Signed narinfos id=55 count=16946worker2 # [ 148.172525] nix-grpc-daemon[1759]: time=2026-09-22T13:47:43.235Z level=INFO msg="Uploading 1 narinfos"6947lb # [ 148.167284] niks3-server[1149]: 2026/09/22 13:47:43 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6948lb # [ 148.173146] niks3-server[1149]: 2026/09/22 13:47:43 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6949lb # [ 148.177865] niks3-server[1149]: 2026/09/22 13:47:43 INFO Completed upload id=556950worker2 # [ 148.190867] nix-grpc-daemon[1759]: time=2026-09-22T13:47:43.253Z level=INFO msg="Upload complete. (250ms)"6951worker2 # [ 148.193121] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53166 duration_s=0 paths=1 nar_bytes_in=5286952worker2 # [ 148.196762] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531666953worker2 # [ 148.203496] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)6954worker2 # [ 148.205934] nix-daemon[870]: reaped child process 1836, status = succeeded6955worker2 # [ 148.211170] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)6956client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6957client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6958worker1 # [ 148.224579] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:42Z level=info event=draining6959client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6960client # farm-a> LOG-farm-a-during-drain6961worker2 # [ 149.449677] nix-daemon[870]: reaped child process 1851, status = succeeded6962worker2 # [ 149.533529] nix-daemon[870]: accepted connection from pid 1867, user nix-grpc-daemon (trusted)6963worker2 # [ 149.549326] nix-daemon[870]: reaped child process 1870, status = succeeded6964lb # [ 149.540595] niks3-server[1149]: 2026/09/22 13:47:44 INFO Received uploads request method=POST path=/api/pending_closures6965worker2 # [ 149.558961] nix-grpc-daemon[1759]: time=2026-09-22T13:47:44.621Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6966worker2 # [ 149.560529] nix-grpc-daemon[1759]: time=2026-09-22T13:47:44.623Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6967lb # [ 149.566827] niks3-server[1149]: 2026/09/22 13:47:44 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6968lb # [ 149.571246] niks3-server[1149]: 2026/09/22 13:47:44 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6969lb # [ 149.578103] niks3-server[1149]: 2026/09/22 13:47:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6970lb # [ 149.581965] niks3-server[1149]: 2026/09/22 13:47:44 INFO Signed narinfos id=56 count=16971worker2 # [ 149.593817] nix-grpc-daemon[1759]: time=2026-09-22T13:47:44.656Z level=INFO msg="Uploading 1 narinfos"6972lb # [ 149.584164] niks3-server[1149]: 2026/09/22 13:47:44 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6973lb # [ 149.592040] niks3-server[1149]: 2026/09/22 13:47:44 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6974lb # [ 149.597514] niks3-server[1149]: 2026/09/22 13:47:44 INFO Completed upload id=566975worker2 # [ 149.609403] nix-grpc-daemon[1759]: time=2026-09-22T13:47:44.672Z level=INFO msg="Upload complete. (190ms)"6976lb # [ 149.601505] niks3-server[1149]: 2026/09/22 13:47:44 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6977worker2 # [ 149.612358] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53166 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=4 outputs=16978worker1 # [ 149.609924] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:43Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06979worker2 # [ 149.620591] nix-daemon[870]: reaped child process 1849, status = succeeded6980worker1 # [ 149.611635] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:43Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16981worker2 # [ 149.662316] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531666982worker2 # [ 149.679993] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:531666983worker2 # [ 149.693273] nix-grpc-daemon[1759]: time=2026-09-22T13:47:44.755Z level=INFO msg="All 1 paths already cached"6984client # copying 1 paths to 'lb:50051'...6985worker2 # [ 149.701967] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:531666986client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6987worker2 # [ 149.719616] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)6988worker2 # [ 149.815980] nix-daemon[870]: accepted connection from pid 1874, user nix-grpc-daemon (trusted)6989worker2 # [ 149.829802] nix-daemon[870]: reaped child process 1877, status = succeeded6990worker2 # [ 149.905025] nix-daemon[870]: accepted connection from pid 1879, user nix-grpc-daemon (trusted)6991worker2 # [ 149.920432] nix-daemon[870]: reaped child process 1882, status = succeeded6992lb # [ 149.912125] niks3-server[1149]: 2026/09/22 13:47:45 INFO Received uploads request method=POST path=/api/pending_closures6993worker2 # [ 149.939899] nix-grpc-daemon[1759]: time=2026-09-22T13:47:45.002Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6994worker2 # [ 149.941975] nix-grpc-daemon[1759]: time=2026-09-22T13:47:45.004Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6995lb # [ 149.945622] niks3-server[1149]: 2026/09/22 13:47:45 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6996lb # [ 149.956716] niks3-server[1149]: 2026/09/22 13:47:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6997lb # [ 149.963262] niks3-server[1149]: 2026/09/22 13:47:45 INFO Signed narinfos id=57 count=16998worker2 # [ 149.977989] nix-grpc-daemon[1759]: time=2026-09-22T13:47:45.040Z level=INFO msg="Uploading 1 narinfos"6999lb # [ 149.970843] niks3-server[1149]: 2026/09/22 13:47:45 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls7000lb # [ 149.982159] niks3-server[1149]: 2026/09/22 13:47:45 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete7001lb # [ 149.987185] niks3-server[1149]: 2026/09/22 13:47:45 INFO Completed upload id=577002worker2 # [ 150.001070] nix-grpc-daemon[1759]: time=2026-09-22T13:47:45.063Z level=INFO msg="Upload complete. (266ms)"7003lb # [ 149.990627] niks3-server[1149]: 2026/09/22 13:47:45 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo7004worker2 # [ 150.005415] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53166 duration_s=0 paths=1 nar_bytes_in=6407005worker2 # [ 150.009927] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531667006worker2 # [ 150.020252] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7007worker2 # [ 150.030121] nix-daemon[870]: reaped child process 1872, status = succeeded7008worker2 # [ 150.032965] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7009client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...7010client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv7011client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7012client # farm-b> LOG-farm-b-during-drain7013worker2 # [ 151.048500] nix-daemon[870]: reaped child process 1886, status = succeeded7014worker2 # [ 151.123501] nix-daemon[870]: accepted connection from pid 1902, user nix-grpc-daemon (trusted)7015worker2 # [ 151.137287] nix-daemon[870]: reaped child process 1905, status = succeeded7016lb # [ 151.133269] niks3-server[1149]: 2026/09/22 13:47:46 INFO Received uploads request method=POST path=/api/pending_closures7017worker2 # [ 151.154102] nix-grpc-daemon[1759]: time=2026-09-22T13:47:46.216Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7018worker2 # [ 151.155596] nix-grpc-daemon[1759]: time=2026-09-22T13:47:46.218Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"7019lb # [ 151.172254] niks3-server[1149]: 2026/09/22 13:47:46 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv7020lb # [ 151.183182] niks3-server[1149]: 2026/09/22 13:47:46 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst7021lb # [ 151.191377] niks3-server[1149]: 2026/09/22 13:47:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign7022worker2 # [ 151.210677] nix-grpc-daemon[1759]: time=2026-09-22T13:47:46.273Z level=INFO msg="Uploading 1 narinfos"7023lb # [ 151.199094] niks3-server[1149]: 2026/09/22 13:47:46 INFO Signed narinfos id=58 count=17024lb # [ 151.210327] niks3-server[1149]: 2026/09/22 13:47:46 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete7025lb # [ 151.214245] niks3-server[1149]: 2026/09/22 13:47:46 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls7026lb # [ 151.219109] niks3-server[1149]: 2026/09/22 13:47:46 INFO Completed upload id=587027worker2 # [ 151.232394] nix-grpc-daemon[1759]: time=2026-09-22T13:47:46.295Z level=INFO msg="Upload complete. (192ms)"7028lb # [ 151.221125] niks3-server[1149]: 2026/09/22 13:47:46 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo7029worker2 # [ 151.235361] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53166 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=5 outputs=17030worker1 # [ 151.232558] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:45Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=07031worker2 # [ 151.243410] nix-daemon[870]: reaped child process 1884, status = succeeded7032worker1 # [ 151.234378] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:45Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=17033worker2 # [ 151.285126] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531667034worker2 # [ 151.293509] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:531667035worker2 # [ 151.302689] nix-grpc-daemon[1759]: time=2026-09-22T13:47:46.365Z level=INFO msg="All 2 paths already cached"7036client # copying 1 paths to 'lb:50051'...7037worker2 # [ 151.308761] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:531667038client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...7039worker2 # [ 151.324451] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7040worker2 # [ 151.411466] nix-daemon[870]: accepted connection from pid 1909, user nix-grpc-daemon (trusted)7041worker2 # [ 151.426045] nix-daemon[870]: reaped child process 1912, status = succeeded7042worker2 # [ 151.506664] nix-daemon[870]: accepted connection from pid 1914, user nix-grpc-daemon (trusted)7043worker2 # [ 151.519827] nix-daemon[870]: reaped child process 1917, status = succeeded7044lb # [ 151.515891] niks3-server[1149]: 2026/09/22 13:47:46 INFO Received uploads request method=POST path=/api/pending_closures7045worker2 # [ 151.535954] nix-grpc-daemon[1759]: time=2026-09-22T13:47:46.598Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7046worker2 # [ 151.537565] nix-grpc-daemon[1759]: time=2026-09-22T13:47:46.600Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"7047lb # [ 151.546810] niks3-server[1149]: 2026/09/22 13:47:46 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst7048lb # [ 151.556418] niks3-server[1149]: 2026/09/22 13:47:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign7049worker2 # [ 151.574244] nix-grpc-daemon[1759]: time=2026-09-22T13:47:46.636Z level=INFO msg="Uploading 1 narinfos"7050lb # [ 151.562294] niks3-server[1149]: 2026/09/22 13:47:46 INFO Signed narinfos id=59 count=17051lb # [ 151.575636] niks3-server[1149]: 2026/09/22 13:47:46 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls7052lb # [ 151.582186] niks3-server[1149]: 2026/09/22 13:47:46 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete7053worker2 # [ 151.599636] nix-grpc-daemon[1759]: time=2026-09-22T13:47:46.662Z level=INFO msg="Upload complete. (267ms)"7054lb # [ 151.587909] niks3-server[1149]: 2026/09/22 13:47:46 INFO Completed upload id=597055worker2 # [ 151.601781] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53166 duration_s=0 paths=1 nar_bytes_in=7687056worker2 # [ 151.605285] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531667057lb # [ 151.594137] niks3-server[1149]: 2026/09/22 13:47:46 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo7058worker2 # [ 151.613484] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7059worker2 # [ 151.617098] nix-daemon[870]: reaped child process 1907, status = succeeded7060worker2 # [ 151.620889] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7061client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...7062client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv7063client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7064client # farm-top> LOG-farm-top-during-drain7065worker2 # [ 152.035127] nix-daemon[870]: reaped child process 1921, status = succeeded7066worker2 # [ 152.105396] nix-daemon[870]: accepted connection from pid 1937, user nix-grpc-daemon (trusted)7067worker2 # [ 152.119729] nix-daemon[870]: reaped child process 1940, status = succeeded7068lb # [ 152.112847] niks3-server[1149]: 2026/09/22 13:47:47 INFO Received uploads request method=POST path=/api/pending_closures7069worker2 # [ 152.133195] nix-grpc-daemon[1759]: time=2026-09-22T13:47:47.195Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7070worker2 # [ 152.134690] nix-grpc-daemon[1759]: time=2026-09-22T13:47:47.197Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"7071lb # [ 152.146601] niks3-server[1149]: 2026/09/22 13:47:47 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv7072lb # [ 152.166335] niks3-server[1149]: 2026/09/22 13:47:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign7073lb # [ 152.170303] niks3-server[1149]: 2026/09/22 13:47:47 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst7074worker2 # [ 152.186338] nix-grpc-daemon[1759]: time=2026-09-22T13:47:47.248Z level=INFO msg="Uploading 1 narinfos"7075lb # [ 152.174740] niks3-server[1149]: 2026/09/22 13:47:47 INFO Signed narinfos id=60 count=17076lb # [ 152.184852] niks3-server[1149]: 2026/09/22 13:47:47 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete7077lb # [ 152.189294] niks3-server[1149]: 2026/09/22 13:47:47 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls7078lb # [ 152.191523] niks3-server[1149]: 2026/09/22 13:47:47 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo7079lb # [ 152.194067] niks3-server[1149]: 2026/09/22 13:47:47 INFO Completed upload id=607080worker2 # [ 152.207468] nix-grpc-daemon[1759]: time=2026-09-22T13:47:47.270Z level=INFO msg="Upload complete. (181ms)"7081worker2 # [ 152.212800] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53166 duration_s=0 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=6 outputs=17082worker2 # [ 152.226902] nix-daemon[870]: reaped child process 1919, status = succeeded7083client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain >&2, in 4.63 seconds)7084client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath7085client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag during-drain outPath, in 0.12 seconds)7086worker1: must succeed: pkill -f 'read -t [9]0 x'7087worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.04 seconds)7088client: waiting for success: ! systemctl is-active sw7089worker1 # [ 152.508583] nix-daemon[848]: reaped child process 1752, status = succeeded7090worker1 # [ 152.580263] nix-daemon[848]: accepted connection from pid 1799, user nix-grpc-daemon (trusted)7091worker1 # [ 152.592335] nix-daemon[848]: reaped child process 1802, status = succeeded7092lb # [ 152.596928] niks3-server[1149]: 2026/09/22 13:47:47 INFO Received uploads request method=POST path=/api/pending_closures7093worker1 # [ 152.607780] nix-grpc-daemon[1684]: time=2026-09-22T13:47:46.880Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7094worker1 # [ 152.609296] nix-grpc-daemon[1684]: time=2026-09-22T13:47:46.882Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"7095lb # [ 152.627660] niks3-server[1149]: 2026/09/22 13:47:47 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv7096lb # [ 152.644956] niks3-server[1149]: 2026/09/22 13:47:47 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst7097lb # [ 152.658882] niks3-server[1149]: 2026/09/22 13:47:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign7098lb # [ 152.662833] niks3-server[1149]: 2026/09/22 13:47:47 INFO Signed narinfos id=61 count=17099worker1 # [ 152.668643] nix-grpc-daemon[1684]: time=2026-09-22T13:47:46.940Z level=INFO msg="Uploading 1 narinfos"7100lb # [ 152.667449] niks3-server[1149]: 2026/09/22 13:47:47 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls7101lb # [ 152.677542] niks3-server[1149]: 2026/09/22 13:47:47 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete7102lb # [ 152.681567] niks3-server[1149]: 2026/09/22 13:47:47 INFO Completed upload id=617103worker1 # [ 152.684384] nix-grpc-daemon[1684]: time=2026-09-22T13:47:46.957Z level=INFO msg="Upload complete. (185ms)"7104lb # [ 152.684499] niks3-server[1149]: 2026/09/22 13:47:47 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo7105worker1 # [ 152.692442] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48452 duration_s=7 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=3 outputs=17106worker1 # [ 152.705881] nix-daemon[848]: reaped child process 1750, status = succeeded7107client # [ 152.745746] systemd[1]: sw.service: Deactivated successfully.7108client # [ 152.750315] systemd[1]: sw.service: Consumed 131ms CPU time over 8.627s wall clock time, 43.4M memory peak, 7.6K incoming IP traffic, 8.6K outgoing IP traffic.7109worker1 # [ 153.226318] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:47Z level=info event=signal_exit7110lb # [ 153.232701] niks3-server[1149]: 2026/09/22 13:47:48 INFO lead: released remote=192.168.1.3:549887111client: (finished: waiting for success: ! systemctl is-active sw, in 1.06 seconds)7112client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }7113client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.03 seconds)7114worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service7115worker2 # [ 154.244985] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:49Z level=info event=scheduler_restarting addr=lb:500517116worker1 # [ 154.240317] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:48Z level=info event=scheduler_restarting7117worker1 # [ 154.244069] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:48Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:484367118worker1 # [ 154.248231] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:48Z level=info event=worker_gone addr=192.168.1.4:500517119worker1 # [ 154.252270] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:48Z level=info event=scheduler_restarting addr=lb:500517120worker1 # [ 154.255599] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:48Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:484367121worker1 # [ 154.259123] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:48Z level=info event=worker_gone addr=192.168.1.3:500517122worker1 # [ 154.274363] nix-daemon[848]: reaped child process 1655, status = succeeded7123worker2 # [ 154.445933] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:49Z level=info event=scheduler_connected addr=lb:500517124worker2 # [ 154.450175] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7125worker1 # [ 154.449447] nix-grpc-daemon[1600]: ts=2026-09-22T13:47:48Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"7126worker2 # [ 154.650817] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:49Z level=info event=scheduler_connected addr=lb:500517127worker2 # [ 154.652664] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7128worker2 # [ 154.853459] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:49Z level=info event=scheduler_connected addr=lb:500517129worker2 # [ 154.855267] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7130worker1 # [ 155.003267] nix-grpc-daemon[1600]: E0000 00:00:1790084869.275801 1646 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 20, started true, shutdown true7131worker1 # [ 155.005249] nix-grpc-daemon[1600]: E0000 00:00:1790084869.275955 1646 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true7132worker2 # [ 155.056108] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:50Z level=info event=scheduler_connected addr=lb:500517133worker2 # [ 155.058042] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7134worker2 # [ 155.259298] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:50Z level=info event=scheduler_connected addr=lb:500517135worker2 # [ 155.260968] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7136worker2 # [ 155.454817] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:50Z level=info event=scheduler_take_over7137lb # [ 155.443498] niks3-server[1149]: 2026/09/22 13:47:50 INFO lead: acquired remote=192.168.1.4:573247138worker2 # [ 155.461604] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:50Z level=info event=scheduler_connected addr=lb:500517139worker2 # [ 155.463241] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7140lb # [ 155.587423] envoy[917]: conn peer=[2001:db8:1::1]:52732 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9568 tx=4381 ms=44647141lb # [ 155.591151] envoy[917]: conn peer=[2001:db8:1::1]:52726 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3018 tx=1558 ms=84697142lb # [ 155.594701] envoy[917]: conn peer=[2001:db8:1::3]:34250 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=958 tx=344 ms=201787143worker2 # [ 155.664588] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:50Z level=info event=scheduler_connected addr=lb:500517144worker2 # [ 155.668536] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7145worker2 # [ 155.869729] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:50Z level=info event=scheduler_connected addr=lb:500517146worker2 # [ 155.873599] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7147worker2 # [ 156.074772] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:51Z level=info event=scheduler_connected addr=lb:500517148worker2 # [ 156.078867] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7149worker1 # [ 156.257484] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.7150worker1 # [ 156.262561] systemd[1]: nix-grpc-daemon.service: Consumed 601ms CPU time over 22.163s wall clock time, 36.8M memory peak, 24.2K incoming IP traffic, 31.4K outgoing IP traffic.7151worker2 # [ 156.279914] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:51Z level=info event=scheduler_connected addr=lb:500517152worker2 # [ 156.281580] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"7153worker1 # [ 156.277168] systemd[1]: Starting Nix worker-protocol over gRPC...7154worker1 # [ 156.365636] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:50Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"7155worker1 # [ 156.369916] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:50Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc7156worker1 # [ 156.372848] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:50Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=17157worker1 # [ 156.376370] nix-grpc-daemon[1816]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR7158worker1 # [ 156.381971] nix-grpc-daemon[1816]: E0000 00:00:1790084870.645185 1816 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.7159worker1 # [ 156.385382] nix-grpc-daemon[1816]: E0000 00:00:1790084870.648899 1816 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.7160worker1 # [ 156.389817] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:50Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket7161worker1 # [ 156.392143] nix-grpc-daemon[1816]: E0000 00:00:1790084870.661146 1857 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.7162worker1 # [ 156.397156] systemd[1]: Started Nix worker-protocol over gRPC.7163worker1 # [ 156.406080] nix-grpc-daemon[1816]: E0000 00:00:1790084870.676664 1857 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.7164worker1 # [ 156.408169] nix-grpc-daemon[1816]: E0000 00:00:1790084870.677116 1857 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.7165worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.89 seconds)7166lb: waiting until exactly no endpoints of x86_64-linux are unhealthy7167lb: must succeed: curl -sf localhost:9901/clusters7168worker1 # [ 156.419742] nix-grpc-daemon[1816]: E0000 00:00:1790084870.691764 1857 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.7169worker1 # [ 156.429584] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:50Z level=info event=scheduler_connected addr=lb:500517170worker2 # [ 156.448487] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:51Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:333467171worker2 # [ 156.450421] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:51Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=07172lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7173worker2 # [ 156.482070] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:51Z level=info event=scheduler_connected addr=lb:500517174worker2 # [ 156.486060] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:51Z level=info event=oidc_jwks provider=mock keys=17175worker2 # [ 156.487790] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:51Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:333467176worker2 # [ 156.489372] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:51Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=07177lb: must succeed: curl -sf localhost:9901/clusters7178lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)7179lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.10 seconds)7180lb: must succeed: curl -sf localhost:9901/clusters7181lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7182worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7183worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7184(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 13.54 seconds)7185subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does7186client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&17187worker2 # [ 157.745755] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:52Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:531667188worker1 # [ 157.748630] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:575547189worker1 # [ 157.759822] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7190worker1 # [ 157.775902] nix-grpc-daemon[1816]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 93 ms (attempt 1/5)7191worker1 # [ 157.876888] nix-grpc-daemon[1816]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 67 ms (attempt 2/5)7192worker1 # [ 157.945905] nix-grpc-daemon[1816]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 128 ms (attempt 3/5)7193worker1 # [ 158.104854] nix-grpc-daemon[1816]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 28 ms (attempt 4/5)7194worker1 # [ 158.134555] nix-grpc-daemon[1816]: disabling binary cache 'https://cache.nixos.org' for 60 seconds7195worker1 # [ 158.137654] nix-grpc-daemon[1816]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org7196worker2 # [ 158.191715] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:53Z level=debug event=want client=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=07197worker2 # [ 158.195954] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:53Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=17198worker2 # [ 158.200285] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:53Z level=debug event=want client=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=07199worker2 # [ 158.204073] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:53Z level=debug event=want client=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=07200worker1 # [ 158.226074] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547201worker1 # [ 158.234984] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:575547202worker1 # [ 158.242291] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:575547203worker1 # [ 158.245914] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7204worker1 # [ 158.257689] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:52Z level=info event=niks3_push_started pid=18767205worker1 # [ 158.346719] nix-grpc-daemon[1876]: time=2026-09-22T13:47:52.619Z level=INFO msg="Configuring client TLS" cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.pem key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/worker1.key ca=""7206worker1 # [ 158.429784] nix-daemon[848]: accepted connection from pid 1889, user nix-grpc-daemon (trusted)7207worker1 # [ 158.442803] nix-daemon[848]: reaped child process 1892, status = succeeded7208worker1 # [ 158.522843] nix-daemon[848]: accepted connection from pid 1894, user nix-grpc-daemon (trusted)7209worker1 # [ 158.535966] nix-daemon[848]: reaped child process 1897, status = succeeded7210lb # [ 158.539984] niks3-server[1149]: 2026/09/22 13:47:53 INFO Received uploads request method=POST path=/api/pending_closures7211worker1 # [ 158.551978] nix-grpc-daemon[1876]: time=2026-09-22T13:47:52.823Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7212worker1 # [ 158.553335] nix-grpc-daemon[1876]: time=2026-09-22T13:47:52.823Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"7213lb # [ 158.577406] niks3-server[1149]: 2026/09/22 13:47:53 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst7214lb # [ 158.590916] niks3-server[1149]: 2026/09/22 13:47:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign7215worker1 # [ 158.600482] nix-grpc-daemon[1876]: time=2026-09-22T13:47:52.872Z level=INFO msg="Uploading 1 narinfos"7216lb # [ 158.597532] niks3-server[1149]: 2026/09/22 13:47:53 INFO Signed narinfos id=62 count=17217lb # [ 158.607574] niks3-server[1149]: 2026/09/22 13:47:53 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls7218lb # [ 158.617847] niks3-server[1149]: 2026/09/22 13:47:53 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete7219lb # [ 158.622120] niks3-server[1149]: 2026/09/22 13:47:53 INFO Completed upload id=627220worker1 # [ 158.624805] nix-grpc-daemon[1876]: time=2026-09-22T13:47:52.897Z level=INFO msg="Upload complete. (275ms)"7221worker1 # [ 158.627143] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=0 paths=1 nar_bytes_in=5047222lb # [ 158.624936] niks3-server[1149]: 2026/09/22 13:47:53 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo7223worker1 # [ 158.630085] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547224worker1 # [ 158.638088] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7225worker1 # [ 158.639375] nix-daemon[848]: reaped child process 1874, status = succeeded7226worker1 # [ 158.644426] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7227worker1 # [ 159.685685] nix-daemon[848]: reaped child process 1901, status = succeeded7228worker1 # [ 159.759661] nix-daemon[848]: accepted connection from pid 1917, user nix-grpc-daemon (trusted)7229worker1 # [ 159.770472] nix-daemon[848]: reaped child process 1920, status = succeeded7230lb # [ 159.775637] niks3-server[1149]: 2026/09/22 13:47:54 INFO Received uploads request method=POST path=/api/pending_closures7231worker1 # [ 159.788782] nix-grpc-daemon[1876]: time=2026-09-22T13:47:54.061Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7232worker1 # [ 159.790334] nix-grpc-daemon[1876]: time=2026-09-22T13:47:54.063Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"7233lb # [ 159.812719] niks3-server[1149]: 2026/09/22 13:47:54 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv7234lb # [ 159.828199] niks3-server[1149]: 2026/09/22 13:47:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign7235lb # [ 159.832862] niks3-server[1149]: 2026/09/22 13:47:54 INFO Signed narinfos id=63 count=17236worker1 # [ 159.838368] nix-grpc-daemon[1876]: time=2026-09-22T13:47:54.110Z level=INFO msg="Uploading 1 narinfos"7237lb # [ 159.841931] niks3-server[1149]: 2026/09/22 13:47:54 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst7238lb # [ 159.847201] niks3-server[1149]: 2026/09/22 13:47:54 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete7239lb # [ 159.850570] niks3-server[1149]: 2026/09/22 13:47:54 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls7240lb # [ 159.853110] niks3-server[1149]: 2026/09/22 13:47:54 INFO Completed upload id=637241worker1 # [ 159.857795] nix-grpc-daemon[1876]: time=2026-09-22T13:47:54.130Z level=INFO msg="Upload complete. (206ms)"7242lb # [ 159.856596] niks3-server[1149]: 2026/09/22 13:47:54 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo7243worker1 # [ 159.862877] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=1 outputs=17244worker2 # [ 159.876309] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:54Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=17245worker1 # [ 159.875652] nix-daemon[848]: reaped child process 1899, status = succeeded7246worker2 # [ 159.883961] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:54Z level=debug event=want client=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=07247worker1 # [ 159.910321] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547248worker1 # [ 159.922642] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:575547249worker1 # [ 159.933468] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:575547250worker1 # [ 159.943165] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7251worker1 # [ 160.036098] nix-daemon[848]: accepted connection from pid 1924, user nix-grpc-daemon (trusted)7252worker1 # [ 160.048592] nix-daemon[848]: reaped child process 1927, status = succeeded7253worker1 # [ 160.128324] nix-daemon[848]: accepted connection from pid 1929, user nix-grpc-daemon (trusted)7254worker1 # [ 160.140846] nix-daemon[848]: reaped child process 1932, status = succeeded7255lb # [ 160.144442] niks3-server[1149]: 2026/09/22 13:47:55 INFO Received uploads request method=POST path=/api/pending_closures7256worker1 # [ 160.155276] nix-grpc-daemon[1876]: time=2026-09-22T13:47:54.427Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7257worker1 # [ 160.157169] nix-grpc-daemon[1876]: time=2026-09-22T13:47:54.429Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"7258lb # [ 160.172210] niks3-server[1149]: 2026/09/22 13:47:55 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst7259lb # [ 160.180728] niks3-server[1149]: 2026/09/22 13:47:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign7260lb # [ 160.185239] niks3-server[1149]: 2026/09/22 13:47:55 INFO Signed narinfos id=64 count=17261worker1 # [ 160.190921] nix-grpc-daemon[1876]: time=2026-09-22T13:47:54.463Z level=INFO msg="Uploading 1 narinfos"7262lb # [ 160.200675] niks3-server[1149]: 2026/09/22 13:47:55 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete7263lb # [ 160.208093] niks3-server[1149]: 2026/09/22 13:47:55 INFO Completed upload id=647264worker1 # [ 160.210839] nix-grpc-daemon[1876]: time=2026-09-22T13:47:54.483Z level=INFO msg="Upload complete. (256ms)"7265worker1 # [ 160.214224] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=0 paths=1 nar_bytes_in=5047266lb # [ 160.214985] niks3-server[1149]: 2026/09/22 13:47:55 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo7267worker1 # [ 160.219127] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547268worker1 # [ 160.227131] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7269lb # [ 160.225655] niks3-server[1149]: 2026/09/22 13:47:55 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls7270worker1 # [ 160.232639] nix-daemon[848]: reaped child process 1922, status = succeeded7271worker1 # [ 160.236675] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7272worker1 # [ 161.651976] nix-daemon[848]: reaped child process 1936, status = succeeded7273worker1 # [ 161.724715] nix-daemon[848]: accepted connection from pid 1952, user nix-grpc-daemon (trusted)7274worker1 # [ 161.737278] nix-daemon[848]: reaped child process 1955, status = succeeded7275lb # [ 161.742276] niks3-server[1149]: 2026/09/22 13:47:56 INFO Received uploads request method=POST path=/api/pending_closures7276worker1 # [ 161.756280] nix-grpc-daemon[1876]: time=2026-09-22T13:47:56.028Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7277worker1 # [ 161.757926] nix-grpc-daemon[1876]: time=2026-09-22T13:47:56.030Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"7278lb # [ 161.776776] niks3-server[1149]: 2026/09/22 13:47:56 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv7279lb # [ 161.795926] niks3-server[1149]: 2026/09/22 13:47:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign7280worker1 # [ 161.804715] nix-grpc-daemon[1876]: time=2026-09-22T13:47:56.077Z level=INFO msg="Uploading 1 narinfos"7281lb # [ 161.801613] niks3-server[1149]: 2026/09/22 13:47:56 INFO Signed narinfos id=65 count=17282lb # [ 161.808177] niks3-server[1149]: 2026/09/22 13:47:56 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst7283lb # [ 161.814297] niks3-server[1149]: 2026/09/22 13:47:56 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls7284lb # [ 161.820286] niks3-server[1149]: 2026/09/22 13:47:56 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete7285lb # [ 161.823376] niks3-server[1149]: 2026/09/22 13:47:56 INFO Completed upload id=657286worker1 # [ 161.826432] nix-grpc-daemon[1876]: time=2026-09-22T13:47:56.099Z level=INFO msg="Upload complete. (182ms)"7287lb # [ 161.825938] niks3-server[1149]: 2026/09/22 13:47:56 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo7288worker1 # [ 161.832739] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=2 outputs=17289worker2 # [ 161.847285] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:56Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=17290worker2 # [ 161.852985] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:56Z level=debug event=want client=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=07291worker1 # [ 161.848159] nix-daemon[848]: reaped child process 1934, status = succeeded7292worker1 # [ 161.878522] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547293worker1 # [ 161.890757] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:575547294worker1 # [ 161.902063] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:575547295worker1 # [ 161.911358] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7296worker1 # [ 162.005900] nix-daemon[848]: accepted connection from pid 1959, user nix-grpc-daemon (trusted)7297worker1 # [ 162.017876] nix-daemon[848]: reaped child process 1962, status = succeeded7298worker1 # [ 162.098155] nix-daemon[848]: accepted connection from pid 1964, user nix-grpc-daemon (trusted)7299worker1 # [ 162.110267] nix-daemon[848]: reaped child process 1967, status = succeeded7300lb # [ 162.114567] niks3-server[1149]: 2026/09/22 13:47:57 INFO Received uploads request method=POST path=/api/pending_closures7301worker1 # [ 162.125258] nix-grpc-daemon[1876]: time=2026-09-22T13:47:56.397Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7302worker1 # [ 162.126680] nix-grpc-daemon[1876]: time=2026-09-22T13:47:56.397Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"7303lb # [ 162.141107] niks3-server[1149]: 2026/09/22 13:47:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7304worker1 # [ 162.150387] nix-grpc-daemon[1876]: time=2026-09-22T13:47:56.422Z level=INFO msg="Uploading 1 narinfos"7305lb # [ 162.147839] niks3-server[1149]: 2026/09/22 13:47:57 INFO Signed narinfos id=66 count=17306lb # [ 162.154657] niks3-server[1149]: 2026/09/22 13:47:57 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst7307lb # [ 162.163423] niks3-server[1149]: 2026/09/22 13:47:57 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls7308lb # [ 162.173931] niks3-server[1149]: 2026/09/22 13:47:57 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7309worker1 # [ 162.182491] nix-grpc-daemon[1876]: time=2026-09-22T13:47:56.455Z level=INFO msg="Upload complete. (259ms)"7310lb # [ 162.179568] niks3-server[1149]: 2026/09/22 13:47:57 INFO Completed upload id=667311worker1 # [ 162.184716] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=0 paths=1 nar_bytes_in=5047312lb # [ 162.186807] niks3-server[1149]: 2026/09/22 13:47:57 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo7313worker1 # [ 162.189267] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547314worker1 # [ 162.197342] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7315worker1 # [ 162.201795] nix-daemon[848]: reaped child process 1957, status = succeeded7316worker1 # [ 162.206541] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7317worker1 # [ 162.618286] nix-daemon[848]: reaped child process 1971, status = succeeded7318worker1 # [ 162.692441] nix-daemon[848]: accepted connection from pid 1987, user nix-grpc-daemon (trusted)7319worker1 # [ 162.704473] nix-daemon[848]: reaped child process 1990, status = succeeded7320lb # [ 162.709187] niks3-server[1149]: 2026/09/22 13:47:57 INFO Received uploads request method=POST path=/api/pending_closures7321worker1 # [ 162.719666] nix-grpc-daemon[1876]: time=2026-09-22T13:47:56.992Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7322worker1 # [ 162.722461] nix-grpc-daemon[1876]: time=2026-09-22T13:47:56.993Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"7323lb # [ 162.736789] niks3-server[1149]: 2026/09/22 13:47:57 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv7324lb # [ 162.755714] niks3-server[1149]: 2026/09/22 13:47:57 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst7325lb # [ 162.763690] niks3-server[1149]: 2026/09/22 13:47:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7326lb # [ 162.767308] niks3-server[1149]: 2026/09/22 13:47:57 INFO Signed narinfos id=67 count=17327worker1 # [ 162.772916] nix-grpc-daemon[1876]: time=2026-09-22T13:47:57.045Z level=INFO msg="Uploading 1 narinfos"7328lb # [ 162.777837] niks3-server[1149]: 2026/09/22 13:47:57 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7329lb # [ 162.786279] niks3-server[1149]: 2026/09/22 13:47:57 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls7330lb # [ 162.790124] niks3-server[1149]: 2026/09/22 13:47:57 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo7331lb # [ 162.792268] niks3-server[1149]: 2026/09/22 13:47:57 INFO Completed upload id=677332worker1 # [ 162.795363] nix-grpc-daemon[1876]: time=2026-09-22T13:47:57.067Z level=INFO msg="Upload complete. (183ms)"7333worker1 # [ 162.800365] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=0 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=3 outputs=17334worker2 # [ 162.810878] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:57Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=17335worker2 # [ 162.820164] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:57Z level=debug event=want client=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=07336worker1 # [ 162.815083] nix-daemon[848]: reaped child process 1969, status = succeeded7337worker1 # [ 162.846581] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547338worker1 # [ 162.857792] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:575547339worker1 # [ 162.870304] nix-grpc-daemon[1876]: time=2026-09-22T13:47:57.142Z level=INFO msg="All 1 paths already cached"7340worker1 # [ 162.877489] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:575547341worker1 # [ 162.885301] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7342worker1 # [ 162.980662] nix-daemon[848]: accepted connection from pid 1994, user nix-grpc-daemon (trusted)7343worker1 # [ 162.993254] nix-daemon[848]: reaped child process 1997, status = succeeded7344worker1 # [ 163.072652] nix-daemon[848]: accepted connection from pid 1999, user nix-grpc-daemon (trusted)7345worker1 # [ 163.084652] nix-daemon[848]: reaped child process 2002, status = succeeded7346lb # [ 163.089187] niks3-server[1149]: 2026/09/22 13:47:58 INFO Received uploads request method=POST path=/api/pending_closures7347worker1 # [ 163.100280] nix-grpc-daemon[1876]: time=2026-09-22T13:47:57.372Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7348worker1 # [ 163.101677] nix-grpc-daemon[1876]: time=2026-09-22T13:47:57.372Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"7349lb # [ 163.113278] niks3-server[1149]: 2026/09/22 13:47:58 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst7350lb # [ 163.126791] niks3-server[1149]: 2026/09/22 13:47:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7351worker1 # [ 163.137719] nix-grpc-daemon[1876]: time=2026-09-22T13:47:57.410Z level=INFO msg="Uploading 1 narinfos"7352lb # [ 163.134294] niks3-server[1149]: 2026/09/22 13:47:58 INFO Signed narinfos id=68 count=17353lb # [ 163.139064] niks3-server[1149]: 2026/09/22 13:47:58 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls7354lb # [ 163.151765] niks3-server[1149]: 2026/09/22 13:47:58 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7355worker1 # [ 163.161257] nix-grpc-daemon[1876]: time=2026-09-22T13:47:57.433Z level=INFO msg="Upload complete. (262ms)"7356lb # [ 163.159171] niks3-server[1149]: 2026/09/22 13:47:58 INFO Completed upload id=687357worker1 # [ 163.163378] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=0 paths=1 nar_bytes_in=6247358lb # [ 163.162994] niks3-server[1149]: 2026/09/22 13:47:58 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo7359worker1 # [ 163.170072] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547360worker1 # [ 163.171788] nix-daemon[848]: reaped child process 1992, status = succeeded7361worker1 # [ 163.174699] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7362worker1 # [ 163.181081] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7363worker1 # [ 164.446590] nix-daemon[848]: reaped child process 2006, status = succeeded7364worker1 # [ 164.488078] nix-daemon[848]: accepted connection from pid 2022, user nix-grpc-daemon (trusted)7365worker1 # [ 164.500651] nix-daemon[848]: reaped child process 2025, status = succeeded7366lb # [ 164.502461] niks3-server[1149]: 2026/09/22 13:47:59 INFO Received uploads request method=POST path=/api/pending_closures7367worker1 # [ 164.513863] nix-grpc-daemon[1876]: time=2026-09-22T13:47:58.786Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7368worker1 # [ 164.515794] nix-grpc-daemon[1876]: time=2026-09-22T13:47:58.788Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"7369lb # [ 164.528901] niks3-server[1149]: 2026/09/22 13:47:59 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst7370lb # [ 164.535874] niks3-server[1149]: 2026/09/22 13:47:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7371lb # [ 164.540669] niks3-server[1149]: 2026/09/22 13:47:59 INFO Signed narinfos id=69 count=17372worker1 # [ 164.545874] nix-grpc-daemon[1876]: time=2026-09-22T13:47:58.818Z level=INFO msg="Uploading 1 narinfos"7373lb # [ 164.547078] niks3-server[1149]: 2026/09/22 13:47:59 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls7374lb # [ 164.552687] niks3-server[1149]: 2026/09/22 13:47:59 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv7375lb # [ 164.557956] niks3-server[1149]: 2026/09/22 13:47:59 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7376lb # [ 164.561290] niks3-server[1149]: 2026/09/22 13:47:59 INFO Completed upload id=697377worker1 # [ 164.564197] nix-grpc-daemon[1876]: time=2026-09-22T13:47:58.836Z level=INFO msg="Upload complete. (175ms)"7378lb # [ 164.563765] niks3-server[1149]: 2026/09/22 13:47:59 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo7379worker1 # [ 164.566854] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=4 outputs=17380worker1 # [ 164.573994] nix-daemon[848]: reaped child process 2004, status = succeeded7381worker2 # [ 164.579658] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:59Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=17382worker2 # [ 164.588709] nix-grpc-daemon[1667]: ts=2026-09-22T13:47:59Z level=debug event=want client=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=07383worker1 # [ 164.613356] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547384worker1 # [ 164.620690] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:575547385worker1 # [ 164.628386] nix-grpc-daemon[1876]: time=2026-09-22T13:47:58.901Z level=INFO msg="All 1 paths already cached"7386worker1 # [ 164.632265] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:575547387worker1 # [ 164.636878] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7388worker1 # [ 164.713742] nix-daemon[848]: accepted connection from pid 2029, user nix-grpc-daemon (trusted)7389worker1 # [ 164.727173] nix-daemon[848]: reaped child process 2032, status = succeeded7390worker1 # [ 164.799318] nix-daemon[848]: accepted connection from pid 2034, user nix-grpc-daemon (trusted)7391worker1 # [ 164.814054] nix-daemon[848]: reaped child process 2037, status = succeeded7392lb # [ 164.810849] niks3-server[1149]: 2026/09/22 13:47:59 INFO Received uploads request method=POST path=/api/pending_closures7393worker1 # [ 164.822346] nix-grpc-daemon[1876]: time=2026-09-22T13:47:59.094Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7394worker1 # [ 164.823740] nix-grpc-daemon[1876]: time=2026-09-22T13:47:59.094Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"7395lb # [ 164.839557] niks3-server[1149]: 2026/09/22 13:47:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7396worker1 # [ 164.849422] nix-grpc-daemon[1876]: time=2026-09-22T13:47:59.122Z level=INFO msg="Uploading 1 narinfos"7397lb # [ 164.846894] niks3-server[1149]: 2026/09/22 13:47:59 INFO Signed narinfos id=70 count=17398lb # [ 164.852326] niks3-server[1149]: 2026/09/22 13:47:59 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls7399lb # [ 164.857120] niks3-server[1149]: 2026/09/22 13:47:59 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst7400lb # [ 164.873089] niks3-server[1149]: 2026/09/22 13:47:59 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7401worker1 # [ 164.880255] nix-grpc-daemon[1876]: time=2026-09-22T13:47:59.152Z level=INFO msg="Upload complete. (238ms)"7402lb # [ 164.877620] niks3-server[1149]: 2026/09/22 13:47:59 INFO Completed upload id=707403worker1 # [ 164.882286] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=0 paths=1 nar_bytes_in=6247404lb # [ 164.882680] niks3-server[1149]: 2026/09/22 13:48:00 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo7405worker1 # [ 164.888686] nix-daemon[848]: reaped child process 2027, status = succeeded7406worker1 # [ 164.890299] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547407worker1 # [ 164.893433] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7408worker1 # [ 164.899610] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7409worker1 # [ 165.531620] nix-daemon[848]: reaped child process 2041, status = succeeded7410worker1 # [ 165.609933] nix-daemon[848]: accepted connection from pid 2057, user nix-grpc-daemon (trusted)7411worker1 # [ 165.623939] nix-daemon[848]: reaped child process 2060, status = succeeded7412lb # [ 165.627914] niks3-server[1149]: 2026/09/22 13:48:00 INFO Received uploads request method=POST path=/api/pending_closures7413worker1 # [ 165.638505] nix-grpc-daemon[1876]: time=2026-09-22T13:47:59.911Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7414worker1 # [ 165.640144] nix-grpc-daemon[1876]: time=2026-09-22T13:47:59.912Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"7415lb # [ 165.662861] niks3-server[1149]: 2026/09/22 13:48:00 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv7416lb # [ 165.677154] niks3-server[1149]: 2026/09/22 13:48:00 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst7417lb # [ 165.682102] niks3-server[1149]: 2026/09/22 13:48:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7418lb # [ 165.686265] niks3-server[1149]: 2026/09/22 13:48:00 INFO Signed narinfos id=71 count=17419worker1 # [ 165.691695] nix-grpc-daemon[1876]: time=2026-09-22T13:47:59.964Z level=INFO msg="Uploading 1 narinfos"7420lb # [ 165.693093] niks3-server[1149]: 2026/09/22 13:48:00 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls7421lb # [ 165.697411] niks3-server[1149]: 2026/09/22 13:48:00 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7422lb # [ 165.700436] niks3-server[1149]: 2026/09/22 13:48:00 INFO Completed upload id=717423worker1 # [ 165.704909] nix-grpc-daemon[1876]: time=2026-09-22T13:47:59.977Z level=INFO msg="Upload complete. (197ms)"7424lb # [ 165.703516] niks3-server[1149]: 2026/09/22 13:48:00 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo7425worker1 # [ 165.711311] nix-grpc-daemon[1816]: ts=2026-09-22T13:47:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=0 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=5 outputs=17426worker2 # [ 165.720886] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:00Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=17427worker1 # [ 165.723271] nix-daemon[848]: reaped child process 2039, status = succeeded7428worker1 # [ 165.755188] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547429worker2 # [ 165.766399] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:00Z level=debug event=want client=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=07430worker1 # [ 165.766701] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:575547431worker1 # [ 165.778112] nix-grpc-daemon[1876]: time=2026-09-22T13:48:00.050Z level=INFO msg="All 1 paths already cached"7432worker1 # [ 165.786131] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:575547433worker1 # [ 165.795171] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7434worker1 # [ 165.891602] nix-daemon[848]: accepted connection from pid 2064, user nix-grpc-daemon (trusted)7435worker1 # [ 165.904897] nix-daemon[848]: reaped child process 2067, status = succeeded7436worker1 # [ 165.983561] nix-daemon[848]: accepted connection from pid 2069, user nix-grpc-daemon (trusted)7437worker1 # [ 165.997692] nix-daemon[848]: reaped child process 2072, status = succeeded7438lb # [ 165.998485] niks3-server[1149]: 2026/09/22 13:48:01 INFO Received uploads request method=POST path=/api/pending_closures7439worker1 # [ 166.009921] nix-grpc-daemon[1876]: time=2026-09-22T13:48:00.282Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7440worker1 # [ 166.011290] nix-grpc-daemon[1876]: time=2026-09-22T13:48:00.282Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"7441lb # [ 166.025149] niks3-server[1149]: 2026/09/22 13:48:01 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst7442lb # [ 166.036570] niks3-server[1149]: 2026/09/22 13:48:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7443worker1 # [ 166.044963] nix-grpc-daemon[1876]: time=2026-09-22T13:48:00.317Z level=INFO msg="Uploading 1 narinfos"7444lb # [ 166.041936] niks3-server[1149]: 2026/09/22 13:48:01 INFO Signed narinfos id=72 count=17445lb # [ 166.047582] niks3-server[1149]: 2026/09/22 13:48:01 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls7446lb # [ 166.056184] niks3-server[1149]: 2026/09/22 13:48:01 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7447lb # [ 166.060323] niks3-server[1149]: 2026/09/22 13:48:01 INFO Completed upload id=727448worker1 # [ 166.063141] nix-grpc-daemon[1876]: time=2026-09-22T13:48:00.335Z level=INFO msg="Upload complete. (257ms)"7449lb # [ 166.062061] niks3-server[1149]: 2026/09/22 13:48:01 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo7450worker1 # [ 166.065368] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=0 paths=1 nar_bytes_in=6247451worker1 # [ 166.069471] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547452worker1 # [ 166.076087] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7453worker1 # [ 166.077340] nix-daemon[848]: reaped child process 2062, status = succeeded7454worker1 # [ 166.082307] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7455worker1 # [ 167.520131] nix-daemon[848]: reaped child process 2076, status = succeeded7456worker1 # [ 167.598878] nix-daemon[848]: accepted connection from pid 2092, user nix-grpc-daemon (trusted)7457worker1 # [ 167.611729] nix-daemon[848]: reaped child process 2095, status = succeeded7458lb # [ 167.615537] niks3-server[1149]: 2026/09/22 13:48:02 INFO Received uploads request method=POST path=/api/pending_closures7459worker1 # [ 167.626230] nix-grpc-daemon[1876]: time=2026-09-22T13:48:01.898Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7460worker1 # [ 167.627887] nix-grpc-daemon[1876]: time=2026-09-22T13:48:01.900Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"7461lb # [ 167.651623] niks3-server[1149]: 2026/09/22 13:48:02 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst7462lb # [ 167.661906] niks3-server[1149]: 2026/09/22 13:48:02 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv7463lb # [ 167.667193] niks3-server[1149]: 2026/09/22 13:48:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7464lb # [ 167.672825] niks3-server[1149]: 2026/09/22 13:48:02 INFO Signed narinfos id=73 count=17465worker1 # [ 167.678630] nix-grpc-daemon[1876]: time=2026-09-22T13:48:01.951Z level=INFO msg="Uploading 1 narinfos"7466lb # [ 167.683155] niks3-server[1149]: 2026/09/22 13:48:02 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls7467lb # [ 167.691615] niks3-server[1149]: 2026/09/22 13:48:02 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7468lb # [ 167.696646] niks3-server[1149]: 2026/09/22 13:48:02 INFO Completed upload id=737469worker1 # [ 167.699846] nix-grpc-daemon[1876]: time=2026-09-22T13:48:01.972Z level=INFO msg="Upload complete. (209ms)"7470lb # [ 167.699394] niks3-server[1149]: 2026/09/22 13:48:02 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo7471worker1 # [ 167.705860] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=6 outputs=17472worker2 # [ 167.716646] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:02Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=17473worker1 # [ 167.715323] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547474worker2 # [ 167.725799] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:02Z level=debug event=want client=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=07475worker1 # [ 167.726062] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:575547476worker1 # [ 167.734583] nix-daemon[848]: reaped child process 2074, status = succeeded7477worker1 # [ 167.737799] nix-grpc-daemon[1876]: time=2026-09-22T13:48:02.009Z level=INFO msg="All 2 paths already cached"7478worker1 # [ 167.741605] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:575547479worker1 # [ 167.746348] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7480worker1 # [ 167.827862] nix-daemon[848]: accepted connection from pid 2099, user nix-grpc-daemon (trusted)7481worker1 # [ 167.843323] nix-daemon[848]: reaped child process 2102, status = succeeded7482worker1 # [ 167.919041] nix-daemon[848]: accepted connection from pid 2104, user nix-grpc-daemon (trusted)7483worker1 # [ 167.932614] nix-daemon[848]: reaped child process 2107, status = succeeded7484lb # [ 167.934616] niks3-server[1149]: 2026/09/22 13:48:03 INFO Received uploads request method=POST path=/api/pending_closures7485worker1 # [ 167.945178] nix-grpc-daemon[1876]: time=2026-09-22T13:48:02.217Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7486worker1 # [ 167.946766] nix-grpc-daemon[1876]: time=2026-09-22T13:48:02.219Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"7487lb # [ 167.965754] niks3-server[1149]: 2026/09/22 13:48:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7488lb # [ 167.972456] niks3-server[1149]: 2026/09/22 13:48:03 INFO Signed narinfos id=74 count=17489worker1 # [ 167.978110] nix-grpc-daemon[1876]: time=2026-09-22T13:48:02.250Z level=INFO msg="Uploading 1 narinfos"7490lb # [ 167.976357] niks3-server[1149]: 2026/09/22 13:48:03 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst7491lb # [ 167.984233] niks3-server[1149]: 2026/09/22 13:48:03 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls7492lb # [ 167.997084] niks3-server[1149]: 2026/09/22 13:48:03 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7493lb # [ 168.002436] niks3-server[1149]: 2026/09/22 13:48:03 INFO Completed upload id=747494worker1 # [ 168.007809] nix-grpc-daemon[1876]: time=2026-09-22T13:48:02.280Z level=INFO msg="Upload complete. (256ms)"7495worker1 # [ 168.010457] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=0 paths=1 nar_bytes_in=7527496lb # [ 168.009581] niks3-server[1149]: 2026/09/22 13:48:03 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo7497worker1 # [ 168.016998] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547498worker1 # [ 168.027130] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7499worker1 # [ 168.030127] nix-daemon[848]: reaped child process 2097, status = succeeded7500worker1 # [ 168.035132] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7501worker1 # [ 168.850843] nix-daemon[848]: reaped child process 2111, status = succeeded7502worker1 # [ 168.923550] nix-daemon[848]: accepted connection from pid 2127, user nix-grpc-daemon (trusted)7503worker1 # [ 168.935760] nix-daemon[848]: reaped child process 2130, status = succeeded7504lb # [ 168.941252] niks3-server[1149]: 2026/09/22 13:48:04 INFO Received uploads request method=POST path=/api/pending_closures7505worker1 # [ 168.952267] nix-grpc-daemon[1876]: time=2026-09-22T13:48:03.224Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7506worker1 # [ 168.953887] nix-grpc-daemon[1876]: time=2026-09-22T13:48:03.226Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"7507lb # [ 168.975144] niks3-server[1149]: 2026/09/22 13:48:04 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv7508lb # [ 168.988761] niks3-server[1149]: 2026/09/22 13:48:04 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst7509lb # [ 169.001408] niks3-server[1149]: 2026/09/22 13:48:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7510lb # [ 169.006340] niks3-server[1149]: 2026/09/22 13:48:04 INFO Signed narinfos id=75 count=17511worker1 # [ 169.011369] nix-grpc-daemon[1876]: time=2026-09-22T13:48:03.283Z level=INFO msg="Uploading 1 narinfos"7512lb # [ 169.012548] niks3-server[1149]: 2026/09/22 13:48:04 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls7513lb # [ 169.021038] niks3-server[1149]: 2026/09/22 13:48:04 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7514lb # [ 169.023461] niks3-server[1149]: 2026/09/22 13:48:04 INFO Completed upload id=757515worker1 # [ 169.026469] nix-grpc-daemon[1876]: time=2026-09-22T13:48:03.299Z level=INFO msg="Upload complete. (184ms)"7516lb # [ 169.026534] niks3-server[1149]: 2026/09/22 13:48:04 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo7517worker1 # [ 169.032915] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=1 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=7 outputs=17518worker2 # [ 169.042740] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:04Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=17519worker1 # [ 169.047246] nix-daemon[848]: reaped child process 2109, status = succeeded7520worker1 # [ 169.083830] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547521worker1 # [ 169.095819] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:575547522worker1 # [ 169.108217] nix-grpc-daemon[1876]: time=2026-09-22T13:48:03.380Z level=INFO msg="All 2 paths already cached"7523worker1 # [ 169.115904] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:575547524worker1 # [ 169.125350] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7525worker1 # [ 169.221069] nix-daemon[848]: accepted connection from pid 2134, user nix-grpc-daemon (trusted)7526worker1 # [ 169.234117] nix-daemon[848]: reaped child process 2137, status = succeeded7527worker1 # [ 169.314772] nix-daemon[848]: accepted connection from pid 2139, user nix-grpc-daemon (trusted)7528worker1 # [ 169.329209] nix-daemon[848]: reaped child process 2142, status = succeeded7529lb # [ 169.329548] niks3-server[1149]: 2026/09/22 13:48:04 INFO Received uploads request method=POST path=/api/pending_closures7530worker1 # [ 169.341036] nix-grpc-daemon[1876]: time=2026-09-22T13:48:03.613Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7531worker1 # [ 169.342548] nix-grpc-daemon[1876]: time=2026-09-22T13:48:03.613Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7532lb # [ 169.358158] niks3-server[1149]: 2026/09/22 13:48:04 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7533lb # [ 169.368994] niks3-server[1149]: 2026/09/22 13:48:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign7534lb # [ 169.373398] niks3-server[1149]: 2026/09/22 13:48:04 INFO Signed narinfos id=76 count=17535worker1 # [ 169.379389] nix-grpc-daemon[1876]: time=2026-09-22T13:48:03.651Z level=INFO msg="Uploading 1 narinfos"7536lb # [ 169.380374] niks3-server[1149]: 2026/09/22 13:48:04 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7537lb # [ 169.393575] niks3-server[1149]: 2026/09/22 13:48:04 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete7538worker1 # [ 169.401614] nix-grpc-daemon[1876]: time=2026-09-22T13:48:03.674Z level=INFO msg="Upload complete. (264ms)"7539lb # [ 169.398740] niks3-server[1149]: 2026/09/22 13:48:04 INFO Completed upload id=767540worker1 # [ 169.403838] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=0 paths=1 nar_bytes_in=7527541lb # [ 169.404109] niks3-server[1149]: 2026/09/22 13:48:04 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7542worker1 # [ 169.409675] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547543worker1 # [ 169.418439] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7544worker1 # [ 169.425079] nix-daemon[848]: reaped child process 2132, status = succeeded7545worker1 # [ 169.427832] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7546worker1 # [ 170.260745] nix-daemon[848]: reaped child process 2146, status = succeeded7547worker1 # [ 170.340561] nix-daemon[848]: accepted connection from pid 2162, user nix-grpc-daemon (trusted)7548worker1 # [ 170.351570] nix-daemon[848]: reaped child process 2165, status = succeeded7549lb # [ 170.356888] niks3-server[1149]: 2026/09/22 13:48:05 INFO Received uploads request method=POST path=/api/pending_closures7550worker1 # [ 170.367904] nix-grpc-daemon[1876]: time=2026-09-22T13:48:04.640Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7551worker1 # [ 170.369439] nix-grpc-daemon[1876]: time=2026-09-22T13:48:04.642Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7552lb # [ 170.392996] niks3-server[1149]: 2026/09/22 13:48:05 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7553lb # [ 170.403286] niks3-server[1149]: 2026/09/22 13:48:05 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7554lb # [ 170.411158] niks3-server[1149]: 2026/09/22 13:48:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign7555lb # [ 170.419798] niks3-server[1149]: 2026/09/22 13:48:05 INFO Signed narinfos id=77 count=17556worker1 # [ 170.425077] nix-grpc-daemon[1876]: time=2026-09-22T13:48:04.697Z level=INFO msg="Uploading 1 narinfos"7557lb # [ 170.428115] niks3-server[1149]: 2026/09/22 13:48:05 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7558lb # [ 170.436392] niks3-server[1149]: 2026/09/22 13:48:05 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete7559lb # [ 170.442443] niks3-server[1149]: 2026/09/22 13:48:05 INFO Completed upload id=777560worker1 # [ 170.445503] nix-grpc-daemon[1876]: time=2026-09-22T13:48:04.718Z level=INFO msg="Upload complete. (210ms)"7561lb # [ 170.445780] niks3-server[1149]: 2026/09/22 13:48:05 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7562worker1 # [ 170.451774] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=8 outputs=17563worker2 # [ 170.461569] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:05Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17564worker1 # [ 170.466472] nix-daemon[848]: reaped child process 2144, status = succeeded7565worker1 # [ 170.503165] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547566worker1 # [ 170.515248] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:575547567worker1 # [ 170.527949] nix-grpc-daemon[1876]: time=2026-09-22T13:48:04.800Z level=INFO msg="All 2 paths already cached"7568worker1 # [ 170.535841] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:575547569worker1 # [ 170.545343] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7570worker1 # [ 170.640859] nix-daemon[848]: accepted connection from pid 2169, user nix-grpc-daemon (trusted)7571worker1 # [ 170.653289] nix-daemon[848]: reaped child process 2172, status = succeeded7572worker1 # [ 170.733711] nix-daemon[848]: accepted connection from pid 2174, user nix-grpc-daemon (trusted)7573worker1 # [ 170.745815] nix-daemon[848]: reaped child process 2177, status = succeeded7574lb # [ 170.750443] niks3-server[1149]: 2026/09/22 13:48:05 INFO Received uploads request method=POST path=/api/pending_closures7575worker1 # [ 170.761202] nix-grpc-daemon[1876]: time=2026-09-22T13:48:05.033Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7576worker1 # [ 170.762613] nix-grpc-daemon[1876]: time=2026-09-22T13:48:05.033Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7577lb # [ 170.778129] niks3-server[1149]: 2026/09/22 13:48:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign7578worker1 # [ 170.792127] nix-grpc-daemon[1876]: time=2026-09-22T13:48:05.064Z level=INFO msg="Uploading 1 narinfos"7579lb # [ 170.790284] niks3-server[1149]: 2026/09/22 13:48:05 INFO Signed narinfos id=78 count=17580lb # [ 170.793698] niks3-server[1149]: 2026/09/22 13:48:05 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7581lb # [ 170.799785] niks3-server[1149]: 2026/09/22 13:48:05 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7582lb # [ 170.810564] niks3-server[1149]: 2026/09/22 13:48:05 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete7583worker1 # [ 170.819808] nix-grpc-daemon[1876]: time=2026-09-22T13:48:05.092Z level=INFO msg="Upload complete. (262ms)"7584lb # [ 170.817161] niks3-server[1149]: 2026/09/22 13:48:05 INFO Completed upload id=787585worker1 # [ 170.822243] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=0 paths=1 nar_bytes_in=7527586lb # [ 170.822243] niks3-server[1149]: 2026/09/22 13:48:05 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7587worker1 # [ 170.829803] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:575547588worker1 # [ 170.839414] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7589worker1 # [ 170.841417] nix-daemon[848]: reaped child process 2167, status = succeeded7590worker1 # [ 170.846673] nix-daemon[848]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)7591worker1 # [ 172.084573] nix-daemon[848]: reaped child process 2181, status = succeeded7592worker1 # [ 172.169299] nix-daemon[848]: accepted connection from pid 2197, user nix-grpc-daemon (trusted)7593worker1 # [ 172.182818] nix-daemon[848]: reaped child process 2200, status = succeeded7594lb # [ 172.186328] niks3-server[1149]: 2026/09/22 13:48:07 INFO Received uploads request method=POST path=/api/pending_closures7595worker1 # [ 172.199265] nix-grpc-daemon[1876]: time=2026-09-22T13:48:06.471Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7596worker1 # [ 172.202055] nix-grpc-daemon[1876]: time=2026-09-22T13:48:06.473Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7597lb # [ 172.224503] niks3-server[1149]: 2026/09/22 13:48:07 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7598lb # [ 172.235798] niks3-server[1149]: 2026/09/22 13:48:07 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7599lb # [ 172.244736] niks3-server[1149]: 2026/09/22 13:48:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign7600lb # [ 172.250942] niks3-server[1149]: 2026/09/22 13:48:07 INFO Signed narinfos id=79 count=17601worker1 # [ 172.256702] nix-grpc-daemon[1876]: time=2026-09-22T13:48:06.529Z level=INFO msg="Uploading 1 narinfos"7602lb # [ 172.257313] niks3-server[1149]: 2026/09/22 13:48:07 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7603lb # [ 172.267596] niks3-server[1149]: 2026/09/22 13:48:07 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete7604lb # [ 172.270072] niks3-server[1149]: 2026/09/22 13:48:07 INFO Completed upload id=797605worker1 # [ 172.273144] nix-grpc-daemon[1876]: time=2026-09-22T13:48:06.545Z level=INFO msg="Upload complete. (218ms)"7606lb # [ 172.272360] niks3-server[1149]: 2026/09/22 13:48:07 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7607worker1 # [ 172.279489] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57554 duration_s=1 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=9 outputs=17608worker1 # [ 172.295782] nix-daemon[848]: reaped child process 2179, status = succeeded7609client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 14.74 seconds)7610worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7611worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7612client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]'7613client # [ 172.431617] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7614client # Running as unit: gpu.service; invocation ID: 3604f1954d9e4b68882071f30754cbb37615client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.06 seconds)7616lb: must succeed: curl -sf localhost:9901/clusters7617lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7618lb: must succeed: curl -sf localhost:9901/clusters7619lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7620worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7621worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7622worker1 # [ 172.545497] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:575547623client # [ 172.576536] nix[1539]: don't know how to build these paths:7624client # [ 172.577418] nix[1539]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7625worker2 # [ 172.565214] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:531667626client # [ 172.600916] nix[1539]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker7627worker2 # [ 172.584178] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:07Z level=debug event=want client=2 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07628lb: must succeed: curl -sf localhost:9901/clusters7629lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7630lb: must succeed: curl -sf localhost:9901/clusters7631lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)7632worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7633worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7634client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7635lb # [ 175.589906] envoy[917]: conn peer=[2001:db8:1::1]:59486 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27051 tx=12290 ms=145617636client # [ 202.603169] nix[1539]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}7637client # [ 202.609572] nix[1539]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7638client # [ 202.643400] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7639client # [ 202.647575] systemd[1]: gpu.service: Failed with result 'exit-code'.7640client # [ 202.650805] systemd[1]: gpu.service: Consumed 157ms CPU time over 30.211s wall clock time, 37.6M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic.7641client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.36 seconds)7642client: must succeed: journalctl -u gpu -o cat7643client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds)7644lb: must succeed: curl -sf localhost:9901/clusters7645lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7646lb: must succeed: curl -sf localhost:9901/clusters7647lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)7648worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7649worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7650(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 45.55 seconds)7651subtest: low disk drains a worker and builds go to the other7652worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7653worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.95 seconds)7654worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7655worker1 # [ 204.393947] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:38Z level=info event=unhealthy reason=min_free7656worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.05 seconds)7657client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&27658worker2 # [ 205.296459] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:531667659client # don't know how to build these paths:7660client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7661worker1 # [ 205.294733] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:575547662worker2 # [ 205.315586] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:40Z level=debug event=want client=3 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07663worker2 # [ 205.318155] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:40Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17664worker2 # [ 205.358582] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531667665worker2 # [ 205.365363] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:531667666client # copying 1 paths to 'lb:50051'...7667client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7668worker2 # [ 205.371387] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:531667669worker2 # [ 205.377330] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7670worker2 # [ 205.462761] nix-daemon[870]: accepted connection from pid 1974, user nix-grpc-daemon (trusted)7671worker2 # [ 205.484941] nix-daemon[870]: reaped child process 1977, status = succeeded7672worker2 # [ 205.559899] nix-daemon[870]: accepted connection from pid 1979, user nix-grpc-daemon (trusted)7673worker2 # [ 205.576316] nix-daemon[870]: reaped child process 1982, status = succeeded7674lb # [ 205.568068] niks3-server[1149]: 2026/09/22 13:48:40 INFO Received uploads request method=POST path=/api/pending_closures7675worker2 # [ 205.586133] nix-grpc-daemon[1759]: time=2026-09-22T13:48:40.648Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7676worker2 # [ 205.587710] nix-grpc-daemon[1759]: time=2026-09-22T13:48:40.648Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7677lb # [ 205.586196] niks3-server[1149]: 2026/09/22 13:48:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign7678lb # [ 205.589218] niks3-server[1149]: 2026/09/22 13:48:40 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7679lb # [ 205.592159] niks3-server[1149]: 2026/09/22 13:48:40 INFO Signed narinfos id=80 count=17680worker2 # [ 205.604031] nix-grpc-daemon[1759]: time=2026-09-22T13:48:40.666Z level=INFO msg="Uploading 1 narinfos"7681lb # [ 205.593744] envoy[917]: conn peer=[2001:db8:1::1]:37302 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=498 ms=300527682lb # [ 205.601532] niks3-server[1149]: 2026/09/22 13:48:40 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete7683lb # [ 205.604096] niks3-server[1149]: 2026/09/22 13:48:40 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7684lb # [ 205.607732] niks3-server[1149]: 2026/09/22 13:48:40 INFO Completed upload id=807685worker2 # [ 205.619589] nix-grpc-daemon[1759]: time=2026-09-22T13:48:40.682Z level=INFO msg="Upload complete. (237ms)"7686worker2 # [ 205.621723] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53166 duration_s=0 paths=1 nar_bytes_in=5127687worker2 # [ 205.625094] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531667688lb # [ 205.615579] niks3-server[1149]: 2026/09/22 13:48:40 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7689worker2 # [ 205.630298] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7690client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7691client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7692worker2 # [ 205.641405] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7693worker2 # [ 205.644226] nix-daemon[870]: reaped child process 1972, status = succeeded7694client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7695client # farm-a> LOG-farm-a-drain7696worker2 # [ 206.462583] nix-daemon[870]: reaped child process 1986, status = succeeded7697worker2 # [ 206.536344] nix-daemon[870]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)7698worker2 # [ 206.550404] nix-daemon[870]: reaped child process 2005, status = succeeded7699lb # [ 206.544594] niks3-server[1149]: 2026/09/22 13:48:41 INFO Received uploads request method=POST path=/api/pending_closures7700worker2 # [ 206.562299] nix-grpc-daemon[1759]: time=2026-09-22T13:48:41.625Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7701worker2 # [ 206.563753] nix-grpc-daemon[1759]: time=2026-09-22T13:48:41.626Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7702lb # [ 206.569051] niks3-server[1149]: 2026/09/22 13:48:41 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7703lb # [ 206.571151] niks3-server[1149]: 2026/09/22 13:48:41 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7704lb # [ 206.575394] niks3-server[1149]: 2026/09/22 13:48:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign7705lb # [ 206.577316] niks3-server[1149]: 2026/09/22 13:48:41 INFO Signed narinfos id=81 count=17706worker2 # [ 206.590614] nix-grpc-daemon[1759]: time=2026-09-22T13:48:41.653Z level=INFO msg="Uploading 1 narinfos"7707lb # [ 206.581875] niks3-server[1149]: 2026/09/22 13:48:41 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7708lb # [ 206.587157] niks3-server[1149]: 2026/09/22 13:48:41 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete7709lb # [ 206.591165] niks3-server[1149]: 2026/09/22 13:48:41 INFO Completed upload id=817710worker2 # [ 206.602987] nix-grpc-daemon[1759]: time=2026-09-22T13:48:41.665Z level=INFO msg="Upload complete. (153ms)"7711lb # [ 206.592431] niks3-server[1149]: 2026/09/22 13:48:41 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7712worker2 # [ 206.605662] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53166 duration_s=0 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=10 outputs=17713worker2 # [ 206.613132] nix-daemon[870]: reaped child process 1984, status = succeeded7714worker2 # [ 206.652624] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:41Z level=debug event=want client=3 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07715worker2 # [ 206.656774] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:41Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17716worker2 # [ 206.701416] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531667717worker2 # [ 206.712287] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:531667718worker2 # [ 206.725725] nix-grpc-daemon[1759]: time=2026-09-22T13:48:41.788Z level=INFO msg="All 1 paths already cached"7719client # copying 1 paths to 'lb:50051'...7720client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7721worker2 # [ 206.733979] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:531667722worker2 # [ 206.744170] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7723worker2 # [ 206.839381] nix-daemon[870]: accepted connection from pid 2011, user nix-grpc-daemon (trusted)7724worker2 # [ 206.853340] nix-daemon[870]: reaped child process 2014, status = succeeded7725worker2 # [ 206.934777] nix-daemon[870]: accepted connection from pid 2016, user nix-grpc-daemon (trusted)7726worker2 # [ 206.948904] nix-daemon[870]: reaped child process 2019, status = succeeded7727lb # [ 206.943199] niks3-server[1149]: 2026/09/22 13:48:42 INFO Received uploads request method=POST path=/api/pending_closures7728worker2 # [ 206.966510] nix-grpc-daemon[1759]: time=2026-09-22T13:48:42.028Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7729worker2 # [ 206.967942] nix-grpc-daemon[1759]: time=2026-09-22T13:48:42.028Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7730lb # [ 206.971158] niks3-server[1149]: 2026/09/22 13:48:42 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7731lb # [ 206.984274] niks3-server[1149]: 2026/09/22 13:48:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign7732lb # [ 206.989089] niks3-server[1149]: 2026/09/22 13:48:42 INFO Signed narinfos id=82 count=17733worker2 # [ 207.003898] nix-grpc-daemon[1759]: time=2026-09-22T13:48:42.066Z level=INFO msg="Uploading 1 narinfos"7734lb # [ 206.995538] niks3-server[1149]: 2026/09/22 13:48:42 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7735lb # [ 207.005068] niks3-server[1149]: 2026/09/22 13:48:42 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete7736lb # [ 207.008817] niks3-server[1149]: 2026/09/22 13:48:42 INFO Completed upload id=827737worker2 # [ 207.020606] nix-grpc-daemon[1759]: time=2026-09-22T13:48:42.083Z level=INFO msg="Upload complete. (265ms)"7738lb # [ 207.010219] niks3-server[1149]: 2026/09/22 13:48:42 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7739worker2 # [ 207.022782] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53166 duration_s=0 paths=1 nar_bytes_in=6247740worker2 # [ 207.025933] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531667741worker2 # [ 207.030449] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7742worker2 # [ 207.037075] nix-daemon[870]: reaped child process 2009, status = succeeded7743worker2 # [ 207.041968] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7744client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7745client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7746client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7747client # farm-b> LOG-farm-b-drain7748worker2 # [ 207.863219] nix-daemon[870]: reaped child process 2023, status = succeeded7749worker2 # [ 207.942585] nix-daemon[870]: accepted connection from pid 2039, user nix-grpc-daemon (trusted)7750worker2 # [ 207.957338] nix-daemon[870]: reaped child process 2042, status = succeeded7751lb # [ 207.951664] niks3-server[1149]: 2026/09/22 13:48:43 INFO Received uploads request method=POST path=/api/pending_closures7752worker2 # [ 207.971920] nix-grpc-daemon[1759]: time=2026-09-22T13:48:43.034Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7753worker2 # [ 207.973493] nix-grpc-daemon[1759]: time=2026-09-22T13:48:43.036Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7754lb # [ 207.992518] niks3-server[1149]: 2026/09/22 13:48:43 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7755lb # [ 208.004967] niks3-server[1149]: 2026/09/22 13:48:43 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7756lb # [ 208.013714] niks3-server[1149]: 2026/09/22 13:48:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign7757lb # [ 208.019217] niks3-server[1149]: 2026/09/22 13:48:43 INFO Signed narinfos id=83 count=17758worker2 # [ 208.033609] nix-grpc-daemon[1759]: time=2026-09-22T13:48:43.096Z level=INFO msg="Uploading 1 narinfos"7759lb # [ 208.030119] niks3-server[1149]: 2026/09/22 13:48:43 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7760lb # [ 208.035276] niks3-server[1149]: 2026/09/22 13:48:43 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete7761lb # [ 208.040251] niks3-server[1149]: 2026/09/22 13:48:43 INFO Completed upload id=837762worker2 # [ 208.052327] nix-grpc-daemon[1759]: time=2026-09-22T13:48:43.114Z level=INFO msg="Upload complete. (197ms)"7763lb # [ 208.044123] niks3-server[1149]: 2026/09/22 13:48:43 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7764worker2 # [ 208.059718] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53166 duration_s=1 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=11 outputs=17765worker2 # [ 208.071577] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:43Z level=debug event=want client=3 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07766worker2 # [ 208.075281] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:43Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17767worker2 # [ 208.085462] nix-daemon[870]: reaped child process 2021, status = succeeded7768worker2 # [ 208.115288] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531667769worker2 # [ 208.128064] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:531667770worker2 # [ 208.141215] nix-grpc-daemon[1759]: time=2026-09-22T13:48:43.203Z level=INFO msg="All 2 paths already cached"7771client # copying 1 paths to 'lb:50051'...7772client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7773worker2 # [ 208.150805] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:531667774worker2 # [ 208.159739] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7775worker2 # [ 208.266353] nix-daemon[870]: accepted connection from pid 2048, user nix-grpc-daemon (trusted)7776worker2 # [ 208.281616] nix-daemon[870]: reaped child process 2051, status = succeeded7777worker2 # [ 208.370173] nix-daemon[870]: accepted connection from pid 2053, user nix-grpc-daemon (trusted)7778worker2 # [ 208.384424] nix-daemon[870]: reaped child process 2056, status = succeeded7779lb # [ 208.378933] niks3-server[1149]: 2026/09/22 13:48:43 INFO Received uploads request method=POST path=/api/pending_closures7780worker2 # [ 208.400350] nix-grpc-daemon[1759]: time=2026-09-22T13:48:43.462Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7781worker2 # [ 208.401757] nix-grpc-daemon[1759]: time=2026-09-22T13:48:43.462Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7782lb # [ 208.400802] niks3-server[1149]: 2026/09/22 13:48:43 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7783lb # [ 208.407285] niks3-server[1149]: 2026/09/22 13:48:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign7784lb # [ 208.412643] niks3-server[1149]: 2026/09/22 13:48:43 INFO Signed narinfos id=84 count=17785worker2 # [ 208.428574] nix-grpc-daemon[1759]: time=2026-09-22T13:48:43.490Z level=INFO msg="Uploading 1 narinfos"7786lb # [ 208.418366] niks3-server[1149]: 2026/09/22 13:48:43 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7787lb # [ 208.424515] niks3-server[1149]: 2026/09/22 13:48:43 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete7788lb # [ 208.428819] niks3-server[1149]: 2026/09/22 13:48:43 INFO Completed upload id=847789worker2 # [ 208.440686] nix-grpc-daemon[1759]: time=2026-09-22T13:48:43.503Z level=INFO msg="Upload complete. (267ms)"7790lb # [ 208.430451] niks3-server[1149]: 2026/09/22 13:48:43 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7791worker2 # [ 208.442917] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53166 duration_s=0 paths=1 nar_bytes_in=7527792worker2 # [ 208.446734] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:531667793worker2 # [ 208.454066] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7794worker2 # [ 208.456523] nix-daemon[870]: reaped child process 2046, status = succeeded7795worker2 # [ 208.461852] nix-daemon[870]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)7796client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7797client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7798client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7799client # farm-top> LOG-farm-top-drain7800worker2 # [ 209.704442] nix-daemon[870]: reaped child process 2060, status = succeeded7801worker2 # [ 209.778515] nix-daemon[870]: accepted connection from pid 2076, user nix-grpc-daemon (trusted)7802worker2 # [ 209.792850] nix-daemon[870]: reaped child process 2079, status = succeeded7803lb # [ 209.787115] niks3-server[1149]: 2026/09/22 13:48:44 INFO Received uploads request method=POST path=/api/pending_closures7804worker2 # [ 209.803936] nix-grpc-daemon[1759]: time=2026-09-22T13:48:44.866Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7805worker2 # [ 209.805430] nix-grpc-daemon[1759]: time=2026-09-22T13:48:44.868Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7806lb # [ 209.822184] niks3-server[1149]: 2026/09/22 13:48:44 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7807lb # [ 209.832149] niks3-server[1149]: 2026/09/22 13:48:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign7808worker2 # [ 209.852279] nix-grpc-daemon[1759]: time=2026-09-22T13:48:44.914Z level=INFO msg="Uploading 1 narinfos"7809lb # [ 209.839886] niks3-server[1149]: 2026/09/22 13:48:44 INFO Signed narinfos id=85 count=17810lb # [ 209.845217] niks3-server[1149]: 2026/09/22 13:48:44 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7811lb # [ 209.852553] niks3-server[1149]: 2026/09/22 13:48:44 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7812lb # [ 209.860721] niks3-server[1149]: 2026/09/22 13:48:44 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete7813lb # [ 209.867581] niks3-server[1149]: 2026/09/22 13:48:44 INFO Completed upload id=857814worker2 # [ 209.880650] nix-grpc-daemon[1759]: time=2026-09-22T13:48:44.943Z level=INFO msg="Upload complete. (210ms)"7815lb # [ 209.869301] niks3-server[1149]: 2026/09/22 13:48:44 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7816worker2 # [ 209.885053] nix-grpc-daemon[1667]: ts=2026-09-22T13:48:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53166 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=12 outputs=17817worker2 # [ 209.893461] nix-daemon[870]: reaped child process 2058, status = succeeded7818client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.pem&client-key=/nix/store/pp86g1qg7pz3xz5b0jg8c12b7nz5nsbd-farm-certs/ci.key' --eval-store auto -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain >&2, in 4.76 seconds)7819client: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath7820client: (finished: must succeed: nix eval --raw -f /nix/store/1v3figrsha1j7bs9k2jhjd61g006bb3c-job.nix --argstr tag drain outPath, in 0.12 seconds)7821worker1: must succeed: rm /nix/.rw-store/fill7822worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.09 seconds)7823worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7824worker1 # [ 210.395359] nix-grpc-daemon[1816]: ts=2026-09-22T13:48:44Z level=info event=healthy reason=min_free7825worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.04 seconds)7826(finished: subtest: low disk drains a worker and builds go to the other, in 8.06 seconds)7827(finished: run the VM test script, in 212.12 seconds)7828test script finished in 212.37s7829cleanup7830kill QemuMachine (pid 49)7831client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7832client # [2026-09-22T13:48:47Z INFO virtiofsd] Client disconnected, shutting down7833client # [2026-09-22T13:48:47Z INFO virtiofsd] Client disconnected, shutting down7834client # [2026-09-22T13:48:47Z INFO virtiofsd] Client disconnected, shutting down7835kill QemuMachine (pid 50)7836lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7837lb # [2026-09-22T13:48:47Z INFO virtiofsd] Client disconnected, shutting down7838lb # [2026-09-22T13:48:47Z INFO virtiofsd] Client disconnected, shutting down7839lb # [2026-09-22T13:48:47Z INFO virtiofsd] Client disconnected, shutting down7840kill QemuMachine (pid 51)7841worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7842worker1 # [2026-09-22T13:48:47Z INFO virtiofsd] Client disconnected, shutting down7843worker1 # [2026-09-22T13:48:47Z INFO virtiofsd] Client disconnected, shutting down7844worker1 # [2026-09-22T13:48:47Z INFO virtiofsd] Client disconnected, shutting down7845kill QemuMachine (pid 52)7846worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/d64q19q1xjdwfhqx6czvrjgrhq0n3lcc-python3-3.14.7/bin/python3.14)7847worker2 # [2026-09-22T13:48:48Z INFO virtiofsd] Client disconnected, shutting down7848worker2 # [2026-09-22T13:48:48Z INFO virtiofsd] Client disconnected, shutting down7849worker2 # [2026-09-22T13:48:48Z INFO virtiofsd] Client disconnected, shutting down7850(finished: cleanup, in 1.08 seconds)7851additionally exposed symbols:7852 client, lb, worker1, worker2,7853 vlan1,7854 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh