vm-test-run-nix-grpc-farm
checks.x86_64-linux.farm
· build #474
· raw
1tribuchet: building on jamie2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12client # Disk image does not exist, creating the virtualisation disk image...13client # Formatting '/build/vm-state-client/tmp.9UuD2ZG9El', fmt=raw size=107374182414client # mke2fs 1.47.4 (6-Mar-2025)15client # Discarding device blocks: 0/262144 done16client: QEMU running (pid 49)17client # Creating filesystem with 262144 4k blocks and 65536 inodes18client # Filesystem UUID: e633dfbf-1afc-450a-99d4-3a91ae67456419client # Superblock backups stored on blocks:20worker2 # Disk image does not exist, creating the virtualisation disk image...21client # 32768, 98304, 163840, 22937622worker2 # Formatting '/build/vm-state-worker2/tmp.oy6XD1KVJ6', fmt=raw size=107374182423client # 24worker2 # mke2fs 1.47.4 (6-Mar-2025)25lb # Disk image does not exist, creating the virtualisation disk image...26client # Allocating group tables: 0/8 done27worker1 # Disk image does not exist, creating the virtualisation disk image...28worker2 # Discarding device blocks: 0/262144 done29lb # Formatting '/build/vm-state-lb/tmp.zYzNTYVmhS', fmt=raw size=107374182430client # Writing inode tables: 0/8 done31worker1 # Formatting '/build/vm-state-worker1/tmp.EgzVbUqF1P', fmt=raw size=107374182432worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes33lb # mke2fs 1.47.4 (6-Mar-2025)34client # Creating journal (8192 blocks): done35worker1 # mke2fs 1.47.4 (6-Mar-2025)36worker2 # Filesystem UUID: 1cba956f-61fe-4621-bf1c-67f0b10fbb8d37lb # Discarding device blocks: 0/262144 done38client # Writing superblocks and filesystem accounting information: 0/8 done39worker1 # Discarding device blocks: 0/262144 done40worker2 # Superblock backups stored on blocks:41lb # Creating filesystem with 262144 4k blocks and 65536 inodes42client # 43worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes44worker2 # 32768, 98304, 163840, 22937645lb # Filesystem UUID: 95d72eab-f798-408f-a4a0-9a641bb5b95046client # Virtualisation disk image created.47worker1 # Filesystem UUID: 011e19e4-05fe-4d6c-8c0f-8d704f71640048worker2 # 49lb # Superblock backups stored on blocks:50worker1: QEMU running (pid 52)51lb: QEMU running (pid 50)52worker2: QEMU running (pid 51)53worker1 # Superblock backups stored on blocks:54lb # 32768, 98304, 163840, 22937655worker1 # 32768, 98304, 163840, 22937656lb # 57(finished: start all VMs, in 0.22 seconds)58worker2 # Allocating group tables: 0/8 done59worker1 # 60lb # Allocating group tables: 0/8 done61client: waiting for unit network-addresses-eth1.service62worker2 # Writing inode tables: 0/8 done63worker1 # Allocating group tables: 0/8 done64lb # Writing inode tables: 0/8 done65client: waiting for the VM to finish booting66worker2 # Creating journal (8192 blocks): done67worker1 # Writing inode tables: 0/8 done68worker2 # Writing superblocks and filesystem accounting information: 0/8 done69worker1 # Creating journal (8192 blocks): done70worker2 # 71worker1 # Writing superblocks and filesystem accounting information: 0/8 done72worker2 # Virtualisation disk image created.73worker1 # 74lb # Creating journal (8192 blocks): done75worker1 # Virtualisation disk image created.76lb # Writing superblocks and filesystem accounting information: 0/8 done77lb # 78lb # Virtualisation disk image created.79worker2 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)80worker1 # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)81client # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)82lb # SeaBIOS (version rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org)83worker2 # 84worker2 # 85worker1 # 86worker1 # 87worker2 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0088worker1 # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA0089worker2 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...90worker2 # 91worker2 # 92worker2 # 93worker2 # 94worker1 # Press Ctrl-B to configure iPXE (PCI 00:02.0)...95worker1 # 96worker1 # 97worker1 # 98worker1 # 99worker2 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00100worker1 # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00101client # 102client # 103worker2 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...104worker2 # 105worker2 # 106client # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00107worker1 # Press Ctrl-B to configure iPXE (PCI 00:08.0)...108worker1 # 109worker1 # 110client # Press Ctrl-B to configure iPXE (PCI 00:02.0)...111client # 112client # 113client # 114client # 115client # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00116client # Press Ctrl-B to configure iPXE (PCI 00:08.0)...117client # 118client # 119lb # 120lb # 121lb # iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+5EFCC730+5EF2C730 CA00122lb # Press Ctrl-B to configure iPXE (PCI 00:02.0)...123lb # 124lb # 125lb # 126lb # 127lb # iPXE (http://ipxe.org) 00:08.0 CB00 PCI2.10 PnP PMM 5EFCC730 5EF2C730 CB00128lb # Press Ctrl-B to configure iPXE (PCI 00:08.0)...129lb # 130lb # 131worker2 # Booting from ROM...132worker1 # Booting from ROM...133client # Booting from ROM...134lb # Booting from ROM...135worker2 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026136worker2 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/475bgh1218crwnxd90vcgf549d3xm0l5-nixos-system-worker2-test/init regInfo=/nix/store/952y3zfhajkahzwaw24ihil46ck3p195-closure-info/registration console=ttyS0,115200n8 console=tty0137worker2 # [ 0.000000] BIOS-provided physical RAM map:138worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable139worker2 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved140worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved141worker2 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable142worker2 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved143worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved144worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved145worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved146client # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026147worker2 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved148worker2 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved149worker2 # [ 0.000000] NX (Execute Disable) protection: active150worker2 # [ 0.000000] APIC: Static calls initialized151worker2 # [ 0.000000] SMBIOS 2.8 present.152client # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/lnpvxnhllsvr2ncl4v29x83d7qqqszn3-nixos-system-client-test/init regInfo=/nix/store/qa6ya6zrr87bipdarghx8a1k6ph447h1-closure-info/registration console=ttyS0,115200n8 console=tty0153client # [ 0.000000] BIOS-provided physical RAM map:154worker2 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014155worker2 # [ 0.000000] DMI: Memory slots populated: 1/1156client # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable157worker2 # [ 0.000000] Hypervisor detected: KVM158client # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved159worker2 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000160client # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved161worker2 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00162worker1 # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026163worker2 # [ 0.000000] kvm-clock: using sched offset of 469206254 cycles164client # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable165client # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved166worker2 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns167client # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved168worker2 # [ 0.000005] tsc: Detected 2400.012 MHz processor169worker2 # [ 0.000810] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000170client # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved171worker1 # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/c29xnmznw4lyy9ki05g66bk4sbx7ky11-nixos-system-worker1-test/init regInfo=/nix/store/3hn4cmks610pg0kl9z9dfk7vz94ihdl3-closure-info/registration console=ttyS0,115200n8 console=tty0172client # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved173worker2 # [ 0.000838] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs174worker1 # [ 0.000000] BIOS-provided physical RAM map:175client # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved176worker2 # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT177worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable178client # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved179worker2 # [ 0.002728] found SMP MP-table at [mem 0x000f5450-0x000f545f]180worker1 # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved181worker2 # [ 0.002739] Using GB pages for direct mapping182client # [ 0.000000] NX (Execute Disable) protection: active183client # [ 0.000000] APIC: Static calls initialized184worker2 # [ 0.002867] RAMDISK: [mem 0x5e367000-0x5ffcffff]185worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved186client # [ 0.000000] SMBIOS 2.8 present.187worker2 # [ 0.002875] ACPI: Early table checksum verification disabled188worker1 # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable189worker2 # [ 0.002877] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )190client # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014191lb # Probing EDD (edd=off to disable)... ok[ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT_DYNAMIC Mon Sep 7 15:23:00 UTC 2026192client # [ 0.000000] DMI: Memory slots populated: 1/1193worker1 # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved194worker2 # [ 0.002881] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)195client # [ 0.000000] Hypervisor detected: KVM196worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved197client # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000198worker2 # [ 0.002885] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)199client # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00200worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved201client # [ 0.000000] kvm-clock: using sched offset of 490042940 cycles202worker2 # [ 0.002892] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)203worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved204worker2 # [ 0.002894] ACPI: FACS 0x000000005FFE0000 000040205worker1 # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved206client # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns207lb # [ 0.000000] Command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/fj20bcn47ir2axl7b7sxwa0fvig2ds2j-nixos-system-lb-test/init regInfo=/nix/store/l5iprc9wdxb7nl0q1bx2grj7psaadl34-closure-info/registration console=ttyS0,115200n8 console=tty0208lb # [ 0.000000] BIOS-provided physical RAM map:209worker2 # [ 0.002895] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)210client # [ 0.000005] tsc: Detected 2400.012 MHz processor211worker1 # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved212lb # [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable213client # [ 0.000804] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000214worker1 # [ 0.000000] NX (Execute Disable) protection: active215worker2 # [ 0.002897] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)216worker1 # [ 0.000000] APIC: Static calls initialized217lb # [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved218client # [ 0.000831] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs219worker1 # [ 0.000000] SMBIOS 2.8 present.220worker2 # [ 0.002898] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)221lb # [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved222client # [ 0.000833] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT223worker1 # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014224lb # [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005ffd7fff] usable225worker2 # [ 0.002900] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)226client # [ 0.002726] found SMP MP-table at [mem 0x000f5450-0x000f545f]227client # [ 0.002737] Using GB pages for direct mapping228worker1 # [ 0.000000] DMI: Memory slots populated: 1/1229lb # [ 0.000000] BIOS-e820: [mem 0x000000005ffd8000-0x000000005fffffff] reserved230worker2 # [ 0.002901] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]231worker1 # [ 0.000000] Hypervisor detected: KVM232client # [ 0.002885] RAMDISK: [mem 0x5e366000-0x5ffcffff]233lb # [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved234worker2 # [ 0.002902] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]235worker1 # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000236client # [ 0.002892] ACPI: Early table checksum verification disabled237worker1 # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00238client # [ 0.002895] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )239lb # [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved240worker2 # [ 0.002903] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]241worker1 # [ 0.000001] kvm-clock: using sched offset of 493485913 cycles242lb # [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved243worker2 # [ 0.002903] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]244client # [ 0.002899] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)245lb # [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved246worker2 # [ 0.002904] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]247worker1 # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns248client # [ 0.002903] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)249lb # [ 0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved250worker1 # [ 0.000005] tsc: Detected 2400.012 MHz processor251worker2 # [ 0.002904] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]252client # [ 0.002909] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)253lb # [ 0.000000] NX (Execute Disable) protection: active254worker1 # [ 0.000812] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000255worker2 # [ 0.002905] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]256lb # [ 0.000000] APIC: Static calls initialized257client # [ 0.002911] ACPI: FACS 0x000000005FFE0000 000040258lb # [ 0.000000] SMBIOS 2.8 present.259worker2 # [ 0.003135] No NUMA configuration found260worker1 # [ 0.000840] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs261client # [ 0.002913] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)262worker2 # [ 0.003136] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]263worker1 # [ 0.000843] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT264lb # [ 0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014265client # [ 0.002914] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)266worker2 # [ 0.003138] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]267lb # [ 0.000000] DMI: Memory slots populated: 1/1268worker2 # [ 0.005561] Zone ranges:269worker1 # [ 0.002735] found SMP MP-table at [mem 0x000f5450-0x000f545f]270lb # [ 0.000000] Hypervisor detected: KVM271worker1 # [ 0.002746] Using GB pages for direct mapping272client # [ 0.002916] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)273worker2 # [ 0.005562] DMA [mem 0x0000000000001000-0x0000000000ffffff]274lb # [ 0.000000] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000275worker1 # [ 0.002892] RAMDISK: [mem 0x5e367000-0x5ffcffff]276worker2 # [ 0.005564] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]277lb # [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00278client # [ 0.002917] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)279worker2 # [ 0.005565] Normal empty280worker1 # [ 0.002901] ACPI: Early table checksum verification disabled281worker2 # [ 0.005566] Device empty282lb # [ 0.000000] kvm-clock: using sched offset of 483610732 cycles283client # [ 0.002919] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]284worker1 # [ 0.002904] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )285worker2 # [ 0.005566] Movable zone start for each node286worker2 # [ 0.005567] Early memory node ranges287lb # [ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns288client # [ 0.002920] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]289worker1 # [ 0.002908] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)290worker2 # [ 0.005567] node 0: [mem 0x0000000000001000-0x000000000009efff]291lb # [ 0.000004] tsc: Detected 2400.012 MHz processor292client # [ 0.002920] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]293worker2 # [ 0.005568] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]294worker1 # [ 0.002912] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)295lb # [ 0.000811] last_pfn = 0x5ffd8 max_arch_pfn = 0x10000000000296client # [ 0.002921] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]297worker2 # [ 0.005569] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]298lb # [ 0.000838] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs299client # [ 0.002921] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]300worker1 # [ 0.002919] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)301worker2 # [ 0.005590] On node 0, zone DMA: 1 pages in unavailable ranges302lb # [ 0.000841] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT303worker1 # [ 0.002921] ACPI: FACS 0x000000005FFE0000 000040304client # [ 0.002922] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]305worker2 # [ 0.005850] On node 0, zone DMA: 97 pages in unavailable ranges306lb # [ 0.002737] found SMP MP-table at [mem 0x000f5450-0x000f545f]307worker1 # [ 0.002923] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)308client # [ 0.002922] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]309lb # [ 0.002747] Using GB pages for direct mapping310worker2 # [ 0.031826] On node 0, zone DMA32: 40 pages in unavailable ranges311client # [ 0.003142] No NUMA configuration found312lb # [ 0.002894] RAMDISK: [mem 0x5e367000-0x5ffcffff]313worker2 # [ 0.032274] ACPI: PM-Timer IO Port: 0x608314worker1 # [ 0.002924] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)315client # [ 0.003143] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]316lb # [ 0.002901] ACPI: Early table checksum verification disabled317worker2 # [ 0.032284] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])318client # [ 0.003145] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]319lb # [ 0.002904] ACPI: RSDP 0x00000000000F5270 000014 (v00 BOCHS )320worker1 # [ 0.002926] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)321client # [ 0.005353] Zone ranges:322worker2 # [ 0.032311] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23323lb # [ 0.002908] ACPI: RSDT 0x000000005FFE2422 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)324client # [ 0.005354] DMA [mem 0x0000000000001000-0x0000000000ffffff]325worker2 # [ 0.032313] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)326worker1 # [ 0.002927] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)327client # [ 0.005356] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]328worker2 # [ 0.032315] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)329lb # [ 0.002912] ACPI: FACP 0x000000005FFE221A 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)330client # [ 0.005357] Normal empty331worker1 # [ 0.002929] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]332client # [ 0.005358] Device empty333worker2 # [ 0.032316] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)334client # [ 0.005358] Movable zone start for each node335worker1 # [ 0.002930] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]336lb # [ 0.002919] ACPI: DSDT 0x000000005FFE0040 0021DA (v01 BOCHS BXPC 00000001 BXPC 00000001)337client # [ 0.005359] Early memory node ranges338worker2 # [ 0.032317] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)339lb # [ 0.002921] ACPI: FACS 0x000000005FFE0000 000040340worker1 # [ 0.002930] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]341client # [ 0.005359] node 0: [mem 0x0000000000001000-0x000000000009efff]342worker2 # [ 0.032317] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)343lb # [ 0.002922] ACPI: APIC 0x000000005FFE230E 000078 (v03 BOCHS BXPC 00000001 BXPC 00000001)344worker1 # [ 0.002931] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]345client # [ 0.005360] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]346worker2 # [ 0.032320] ACPI: Using ACPI (MADT) for SMP configuration information347lb # [ 0.002924] ACPI: HPET 0x000000005FFE2386 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)348client # [ 0.005361] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]349worker1 # [ 0.002931] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]350worker2 # [ 0.032321] ACPI: HPET id: 0x8086a201 base: 0xfed00000351worker2 # [ 0.032324] TSC deadline timer available352client # [ 0.005381] On node 0, zone DMA: 1 pages in unavailable ranges353lb # [ 0.002925] ACPI: MCFG 0x000000005FFE23BE 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)354worker1 # [ 0.002932] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]355worker2 # [ 0.032328] CPU topo: Max. logical packages: 1356client # [ 0.005642] On node 0, zone DMA: 97 pages in unavailable ranges357worker2 # [ 0.032329] CPU topo: Max. logical dies: 1358worker1 # [ 0.002932] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]359lb # [ 0.002927] ACPI: WAET 0x000000005FFE23FA 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)360client # [ 0.031796] On node 0, zone DMA32: 40 pages in unavailable ranges361worker2 # [ 0.032330] CPU topo: Max. dies per package: 1362worker1 # [ 0.003191] No NUMA configuration found363client # [ 0.032250] ACPI: PM-Timer IO Port: 0x608364lb # [ 0.002928] ACPI: Reserving FACP table memory at [mem 0x5ffe221a-0x5ffe230d]365worker2 # [ 0.032333] CPU topo: Max. threads per core: 1366worker1 # [ 0.003192] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]367client # [ 0.032261] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])368worker2 # [ 0.032334] CPU topo: Num. cores per package: 1369lb # [ 0.002929] ACPI: Reserving DSDT table memory at [mem 0x5ffe0040-0x5ffe2219]370worker1 # [ 0.003194] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]371client # [ 0.032289] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23372worker2 # [ 0.032334] CPU topo: Num. threads per package: 1373worker1 # [ 0.005640] Zone ranges:374lb # [ 0.002930] ACPI: Reserving FACS table memory at [mem 0x5ffe0000-0x5ffe003f]375worker2 # [ 0.032334] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs376client # [ 0.032292] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)377worker1 # [ 0.005641] DMA [mem 0x0000000000001000-0x0000000000ffffff]378lb # [ 0.002930] ACPI: Reserving APIC table memory at [mem 0x5ffe230e-0x5ffe2385]379client # [ 0.032293] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)380worker2 # [ 0.032350] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()381worker1 # [ 0.005642] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]382lb # [ 0.002930] ACPI: Reserving HPET table memory at [mem 0x5ffe2386-0x5ffe23bd]383worker1 # [ 0.005644] Normal empty384client # [ 0.032295] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)385worker1 # [ 0.005644] Device empty386worker2 # [ 0.032380] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]387lb # [ 0.002931] ACPI: Reserving MCFG table memory at [mem 0x5ffe23be-0x5ffe23f9]388worker1 # [ 0.005645] Movable zone start for each node389client # [ 0.032295] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)390worker2 # [ 0.032382] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]391worker1 # [ 0.005646] Early memory node ranges392lb # [ 0.002932] ACPI: Reserving WAET table memory at [mem 0x5ffe23fa-0x5ffe2421]393client # [ 0.032296] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)394lb # [ 0.003154] No NUMA configuration found395worker2 # [ 0.032383] [mem 0x60000000-0xafffffff] available for PCI devices396worker1 # [ 0.005646] node 0: [mem 0x0000000000001000-0x000000000009efff]397client # [ 0.032299] ACPI: Using ACPI (MADT) for SMP configuration information398worker2 # [ 0.032384] Booting paravirtualized kernel on KVM399lb # [ 0.003155] Faking a node at [mem 0x0000000000000000-0x000000005ffd7fff]400worker1 # [ 0.005647] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]401client # [ 0.032300] ACPI: HPET id: 0x8086a201 base: 0xfed00000402lb # [ 0.003157] NODE_DATA(0) allocated [mem 0x5ffd2780-0x5ffd7cff]403client # [ 0.032303] TSC deadline timer available404lb # [ 0.005374] Zone ranges:405worker2 # [ 0.032387] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns406worker1 # [ 0.005648] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]407client # [ 0.032307] CPU topo: Max. logical packages: 1408lb # [ 0.005375] DMA [mem 0x0000000000001000-0x0000000000ffffff]409worker1 # [ 0.005673] On node 0, zone DMA: 1 pages in unavailable ranges410client # [ 0.032308] CPU topo: Max. logical dies: 1411worker2 # [ 0.036821] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1412lb # [ 0.005377] DMA32 [mem 0x0000000001000000-0x000000005ffd7fff]413client # [ 0.032308] CPU topo: Max. dies per package: 1414worker1 # [ 0.005960] On node 0, zone DMA: 97 pages in unavailable ranges415lb # [ 0.005378] Normal empty416worker2 # [ 0.039239] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152417lb # [ 0.005379] Device empty418client # [ 0.032311] CPU topo: Max. threads per core: 1419worker1 # [ 0.034632] On node 0, zone DMA32: 40 pages in unavailable ranges420worker2 # [ 0.039282] kvm-guest: PV spinlocks disabled, single CPU421client # [ 0.032312] CPU topo: Num. cores per package: 1422lb # [ 0.005379] Movable zone start for each node423worker1 # [ 0.035142] ACPI: PM-Timer IO Port: 0x608424lb # [ 0.005380] Early memory node ranges425client # [ 0.032312] CPU topo: Num. threads per package: 1426worker1 # [ 0.035155] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])427lb # [ 0.005381] node 0: [mem 0x0000000000001000-0x000000000009efff]428client # [ 0.032313] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs429worker1 # [ 0.035186] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23430lb # [ 0.005382] node 0: [mem 0x0000000000100000-0x000000005ffd7fff]431client # [ 0.032331] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()432worker1 # [ 0.035189] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)433lb # [ 0.005383] Initmem setup node 0 [mem 0x0000000000001000-0x000000005ffd7fff]434client # [ 0.032362] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]435lb # [ 0.005402] On node 0, zone DMA: 1 pages in unavailable ranges436worker1 # [ 0.035190] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)437worker2 # [ 0.039283] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/475bgh1218crwnxd90vcgf549d3xm0l5-nixos-system-worker2-test/init regInfo=/nix/store/952y3zfhajkahzwaw24ihil46ck3p195-closure-info/registration console=ttyS0,115200n8 console=tty0438client # [ 0.032364] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]439lb # [ 0.005662] On node 0, zone DMA: 97 pages in unavailable ranges440worker1 # [ 0.035191] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)441client # [ 0.032365] [mem 0x60000000-0xafffffff] available for PCI devices442lb # [ 0.031675] On node 0, zone DMA32: 40 pages in unavailable ranges443worker1 # [ 0.035192] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)444client # [ 0.032367] Booting paravirtualized kernel on KVM445lb # [ 0.032128] ACPI: PM-Timer IO Port: 0x608446worker2 # [ 0.039374] Unknown kernel command line parameters "regInfo=/nix/store/952y3zfhajkahzwaw24ihil46ck3p195-closure-info/registration", will be passed to user space.447worker2 # [ 0.039386] random: crng init done448lb # [ 0.032138] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])449worker1 # [ 0.035193] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)450client # [ 0.032370] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns451lb # [ 0.032170] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23452worker2 # [ 0.039387] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes453worker1 # [ 0.035195] ACPI: Using ACPI (MADT) for SMP configuration information454client # [ 0.036830] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1455lb # [ 0.032172] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)456worker1 # [ 0.035197] ACPI: HPET id: 0x8086a201 base: 0xfed00000457worker2 # [ 0.041459] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)458worker1 # [ 0.035199] TSC deadline timer available459client # [ 0.039108] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152460lb # [ 0.032174] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)461worker1 # [ 0.035203] CPU topo: Max. logical packages: 1462worker2 # [ 0.042508] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)463client # [ 0.039147] kvm-guest: PV spinlocks disabled, single CPU464lb # [ 0.032175] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)465worker1 # [ 0.035204] CPU topo: Max. logical dies: 1466worker2 # [ 0.042540] Fallback order for Node 0: 0467worker1 # [ 0.035205] CPU topo: Max. dies per package: 1468lb # [ 0.032176] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)469worker2 # [ 0.042543] Built 1 zonelists, mobility grouping on. Total pages: 393078470worker1 # [ 0.035208] CPU topo: Max. threads per core: 1471worker2 # [ 0.042544] Policy zone: DMA32472lb # [ 0.032177] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)473worker1 # [ 0.035208] CPU topo: Num. cores per package: 1474worker2 # [ 0.045146] mem auto-init: stack:all(zero), heap alloc:on, heap free:off475lb # [ 0.032179] ACPI: Using ACPI (MADT) for SMP configuration information476worker1 # [ 0.035209] CPU topo: Num. threads per package: 1477worker2 # [ 0.048165] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1478lb # [ 0.032180] ACPI: HPET id: 0x8086a201 base: 0xfed00000479client # [ 0.039149] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/lnpvxnhllsvr2ncl4v29x83d7qqqszn3-nixos-system-client-test/init regInfo=/nix/store/qa6ya6zrr87bipdarghx8a1k6ph447h1-closure-info/registration console=ttyS0,115200n8 console=tty0480worker1 # [ 0.035209] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs481worker2 # [ 0.051471] allocated 3145728 bytes of page_ext482lb # [ 0.032183] TSC deadline timer available483lb # [ 0.032188] CPU topo: Max. logical packages: 1484worker2 # [ 0.061318] ftrace: allocating 48731 entries in 192 pages485worker1 # [ 0.035228] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()486lb # [ 0.032189] CPU topo: Max. logical dies: 1487worker2 # [ 0.061320] ftrace: allocated 192 pages with 2 groups488client # [ 0.039241] Unknown kernel command line parameters "regInfo=/nix/store/qa6ya6zrr87bipdarghx8a1k6ph447h1-closure-info/registration", will be passed to user space.489lb # [ 0.032189] CPU topo: Max. dies per package: 1490worker2 # [ 0.062129] Dynamic Preempt: lazy491worker1 # [ 0.035260] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]492client # [ 0.039254] random: crng init done493lb # [ 0.032193] CPU topo: Max. threads per core: 1494worker2 # [ 0.062283] rcu: Preemptible hierarchical RCU implementation.495worker1 # [ 0.035262] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]496lb # [ 0.032193] CPU topo: Num. cores per package: 1497client # [ 0.039254] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes498worker2 # [ 0.062284] rcu: RCU event tracing is enabled.499lb # [ 0.032194] CPU topo: Num. threads per package: 1500worker1 # [ 0.035263] [mem 0x60000000-0xafffffff] available for PCI devices501client # [ 0.041339] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)502worker2 # [ 0.062285] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.503worker1 # [ 0.035265] Booting paravirtualized kernel on KVM504lb # [ 0.032194] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs505worker2 # [ 0.062286] Trampoline variant of Tasks RCU enabled.506client # [ 0.042380] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)507lb # [ 0.032211] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()508worker2 # [ 0.062287] Rude variant of Tasks RCU enabled.509client # [ 0.042411] Fallback order for Node 0: 0510worker1 # [ 0.035267] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns511worker2 # [ 0.062287] Tracing variant of Tasks RCU enabled.512lb # [ 0.032242] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]513client # [ 0.042414] Built 1 zonelists, mobility grouping on. Total pages: 393078514worker1 # [ 0.039741] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1515client # [ 0.042415] Policy zone: DMA32516worker2 # [ 0.062288] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.517lb # [ 0.032244] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]518worker1 # [ 0.042036] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152519client # [ 0.044837] mem auto-init: stack:all(zero), heap alloc:on, heap free:off520lb # [ 0.032245] [mem 0x60000000-0xafffffff] available for PCI devices521worker2 # [ 0.062289] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1522worker1 # [ 0.042085] kvm-guest: PV spinlocks disabled, single CPU523client # [ 0.047869] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1524lb # [ 0.032246] Booting paravirtualized kernel on KVM525client # [ 0.051275] allocated 3145728 bytes of page_ext526worker2 # [ 0.062310] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.527client # [ 0.061135] ftrace: allocating 48731 entries in 192 pages528lb # [ 0.032249] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns529worker2 # [ 0.062311] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.530client # [ 0.061137] ftrace: allocated 192 pages with 2 groups531lb # [ 0.036670] setup_percpu: NR_CPUS:384 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1532client # [ 0.061950] Dynamic Preempt: lazy533worker2 # [ 0.062312] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.534client # [ 0.062073] rcu: Preemptible hierarchical RCU implementation.535lb # [ 0.038769] percpu: Embedded 98 pages/cpu s278528 r8192 d114688 u2097152536client # [ 0.062074] rcu: RCU event tracing is enabled.537lb # [ 0.038811] kvm-guest: PV spinlocks disabled, single CPU538worker2 # [ 0.066633] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16539worker1 # [ 0.042086] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/c29xnmznw4lyy9ki05g66bk4sbx7ky11-nixos-system-worker1-test/init regInfo=/nix/store/3hn4cmks610pg0kl9z9dfk7vz94ihdl3-closure-info/registration console=ttyS0,115200n8 console=tty0540client # [ 0.062074] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.541worker2 # [ 0.066904] rcu: srcu_init: Setting srcu_struct sizes based on contention.542client # [ 0.062076] Trampoline variant of Tasks RCU enabled.543client # [ 0.062076] Rude variant of Tasks RCU enabled.544worker2 # [ 0.066910] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns545worker1 # [ 0.042181] Unknown kernel command line parameters "regInfo=/nix/store/3hn4cmks610pg0kl9z9dfk7vz94ihdl3-closure-info/registration", will be passed to user space.546client # [ 0.062077] Tracing variant of Tasks RCU enabled.547worker1 # [ 0.042194] random: crng init done548worker2 # [ 0.067012] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)549client # [ 0.062077] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.550worker1 # [ 0.042195] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes551lb # [ 0.038812] Kernel command line: console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/fj20bcn47ir2axl7b7sxwa0fvig2ds2j-nixos-system-lb-test/init regInfo=/nix/store/l5iprc9wdxb7nl0q1bx2grj7psaadl34-closure-info/registration console=ttyS0,115200n8 console=tty0552worker2 # [ 0.070522] Console: colour VGA+ 80x25553client # [ 0.062078] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1554worker2 # [ 0.070524] printk: legacy console [tty0] enabled555worker1 # [ 0.044481] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)556worker2 # [ 0.100594] printk: legacy console [ttyS0] enabled557client # [ 0.062134] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.558lb # [ 0.038903] Unknown kernel command line parameters "regInfo=/nix/store/l5iprc9wdxb7nl0q1bx2grj7psaadl34-closure-info/registration", will be passed to user space.559worker1 # [ 0.045608] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)560worker2 # [ 0.200747] ACPI: Core revision 20250807561lb # [ 0.038916] random: crng init done562worker1 # [ 0.045639] Fallback order for Node 0: 0563client # [ 0.062136] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.564worker1 # [ 0.045642] Built 1 zonelists, mobility grouping on. Total pages: 393078565lb # [ 0.038916] printk: log buffer data + meta data: 262144 + 917504 = 1179648 bytes566worker1 # [ 0.045643] Policy zone: DMA32567client # [ 0.062137] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.568worker2 # [ 0.201659] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns569lb # [ 0.040988] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)570client # [ 0.066465] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16571worker1 # [ 0.048562] mem auto-init: stack:all(zero), heap alloc:on, heap free:off572worker2 # [ 0.203210] APIC: Switch to symmetric I/O mode setup573lb # [ 0.042204] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)574client # [ 0.066745] rcu: srcu_init: Setting srcu_struct sizes based on contention.575worker1 # [ 0.051884] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1576lb # [ 0.042239] Fallback order for Node 0: 0577worker2 # [ 0.204207] x2apic enabled578worker1 # [ 0.055439] allocated 3145728 bytes of page_ext579lb # [ 0.042242] Built 1 zonelists, mobility grouping on. Total pages: 393078580client # [ 0.066751] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns581lb # [ 0.042243] Policy zone: DMA32582worker2 # [ 0.204997] APIC: Switched APIC routing to: physical x2apic583worker1 # [ 0.065416] ftrace: allocating 48731 entries in 192 pages584worker1 # [ 0.065418] ftrace: allocated 192 pages with 2 groups585lb # [ 0.044653] mem auto-init: stack:all(zero), heap alloc:on, heap free:off586client # [ 0.066856] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)587worker1 # [ 0.066305] Dynamic Preempt: lazy588client # [ 0.070363] Console: colour VGA+ 80x25589lb # [ 0.047606] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1590worker1 # [ 0.066499] rcu: Preemptible hierarchical RCU implementation.591client # [ 0.070366] printk: legacy console [tty0] enabled592worker2 # [ 0.206873] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1593lb # [ 0.050811] allocated 3145728 bytes of page_ext594worker1 # [ 0.066500] rcu: RCU event tracing is enabled.595client # [ 0.099684] printk: legacy console [ttyS0] enabled596lb # [ 0.060697] ftrace: allocating 48731 entries in 192 pages597worker1 # [ 0.066500] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.598client # [ 0.195430] ACPI: Core revision 20250807599lb # [ 0.060699] ftrace: allocated 192 pages with 2 groups600lb # [ 0.061529] Dynamic Preempt: lazy601worker2 # [ 0.207882] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns602worker1 # [ 0.066502] Trampoline variant of Tasks RCU enabled.603lb # [ 0.061658] rcu: Preemptible hierarchical RCU implementation.604worker1 # [ 0.066503] Rude variant of Tasks RCU enabled.605client # [ 0.196267] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns606lb # [ 0.061659] rcu: RCU event tracing is enabled.607worker1 # [ 0.066503] Tracing variant of Tasks RCU enabled.608worker2 # [ 0.209569] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)609lb # [ 0.061659] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.610client # [ 0.197756] APIC: Switch to symmetric I/O mode setup611worker1 # [ 0.066504] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.612lb # [ 0.061660] Trampoline variant of Tasks RCU enabled.613worker2 # [ 0.210660] x86/cpu: User Mode Instruction Prevention (UMIP) activated614client # [ 0.198692] x2apic enabled615lb # [ 0.061661] Rude variant of Tasks RCU enabled.616worker1 # [ 0.066505] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1617worker2 # [ 0.212565] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127618lb # [ 0.061661] Tracing variant of Tasks RCU enabled.619client # [ 0.199427] APIC: Switched APIC routing to: physical x2apic620worker1 # [ 0.066526] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.621lb # [ 0.061662] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.622worker2 # [ 0.213546] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0623lb # [ 0.061663] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1624worker1 # [ 0.066528] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.625client # [ 0.201216] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1626lb # [ 0.061721] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.627worker2 # [ 0.214570] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto628worker1 # [ 0.066529] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.629client # [ 0.202181] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns630worker1 # [ 0.070939] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16631lb # [ 0.061723] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.632worker2 # [ 0.215566] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl633worker1 # [ 0.071221] rcu: srcu_init: Setting srcu_struct sizes based on contention.634lb # [ 0.061724] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.635client # [ 0.203788] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)636worker2 # [ 0.217566] Transient Scheduler Attacks: Vulnerable: No microcode637lb # [ 0.066246] NR_IRQS: 24832, nr_irqs: 256, preallocated irqs: 16638worker1 # [ 0.071228] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns639client # [ 0.205867] x86/cpu: User Mode Instruction Prevention (UMIP) activated640worker2 # [ 0.218565] Spectre V2 : Mitigation: Enhanced / Automatic IBRS641lb # [ 0.066542] rcu: srcu_init: Setting srcu_struct sizes based on contention.642worker1 # [ 0.071334] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)643client # [ 0.206956] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127644worker2 # [ 0.219506] Speculative Return Stack Overflow: Mitigation: Safe RET645lb # [ 0.066549] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns646worker1 # [ 0.075265] Console: colour VGA+ 80x25647worker1 # [ 0.075268] printk: legacy console [tty0] enabled648client # [ 0.207781] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0649lb # [ 0.066658] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)650worker1 # [ 0.105899] printk: legacy console [ttyS0] enabled651lb # [ 0.070138] Console: colour VGA+ 80x25652worker1 # [ 0.208533] ACPI: Core revision 20250807653lb # [ 0.070141] printk: legacy console [tty0] enabled654client # [ 0.208784] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto655lb # [ 0.099491] printk: legacy console [ttyS0] enabled656lb # [ 0.194721] ACPI: Core revision 20250807657client # [ 0.209781] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl658worker2 # [ 0.220566] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization659worker1 # [ 0.209481] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns660worker2 # [ 0.222573] Spectre V2 : Enabling IBPB for BPF661client # [ 0.210781] Transient Scheduler Attacks: Vulnerable: No microcode662worker1 # [ 0.211061] APIC: Switch to symmetric I/O mode setup663lb # [ 0.195595] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns664worker1 # [ 0.212097] x2apic enabled665client # [ 0.211780] Spectre V2 : Mitigation: Enhanced / Automatic IBRS666lb # [ 0.197060] APIC: Switch to symmetric I/O mode setup667worker2 # [ 0.223296] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier668client # [ 0.212781] Speculative Return Stack Overflow: Mitigation: Safe RET669worker1 # [ 0.212852] APIC: Switched APIC routing to: physical x2apic670lb # [ 0.198041] x2apic enabled671worker2 # [ 0.224566] active return thunk: srso_alias_return_thunk672lb # [ 0.198803] APIC: Switched APIC routing to: physical x2apic673worker2 # [ 0.225587] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'674worker1 # [ 0.214763] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1675worker2 # [ 0.226565] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'676lb # [ 0.200617] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1677worker2 # [ 0.227565] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'678worker1 # [ 0.215846] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns679client # [ 0.213780] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization680client # [ 0.215788] Spectre V2 : Enabling IBPB for BPF681lb # [ 0.201611] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns682worker2 # [ 0.228566] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'683worker1 # [ 0.217583] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)684worker2 # [ 0.229566] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'685client # [ 0.216782] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier686lb # [ 0.203226] Calibrating delay loop (skipped) preset value.. 4800.02 BogoMIPS (lpj=2400012)687worker1 # [ 0.219583] x86/cpu: User Mode Instruction Prevention (UMIP) activated688worker2 # [ 0.230565] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'689client # [ 0.217781] active return thunk: srso_alias_return_thunk690lb # [ 0.204310] x86/cpu: User Mode Instruction Prevention (UMIP) activated691worker2 # [ 0.231566] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'692worker1 # [ 0.221111] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127693lb # [ 0.205391] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127694client # [ 0.218802] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'695worker1 # [ 0.221580] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0696worker2 # [ 0.233490] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256697client # [ 0.219781] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'698lb # [ 0.206224] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0699worker2 # [ 0.234280] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64700client # [ 0.220781] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'701worker1 # [ 0.222584] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto702lb # [ 0.207227] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto703worker2 # [ 0.235289] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512704client # [ 0.221781] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'705worker2 # [ 0.236289] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024706client # [ 0.222781] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'707lb # [ 0.209223] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl708worker1 # [ 0.224580] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl709worker2 # [ 0.237280] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8710client # [ 0.223781] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'711lb # [ 0.210224] Transient Scheduler Attacks: Vulnerable: No microcode712worker1 # [ 0.225580] Transient Scheduler Attacks: Vulnerable: No microcode713lb # [ 0.211223] Spectre V2 : Mitigation: Enhanced / Automatic IBRS714worker1 # [ 0.227349] Spectre V2 : Mitigation: Enhanced / Automatic IBRS715client # [ 0.224781] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'716worker2 # [ 0.238295] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.717lb # [ 0.212223] Speculative Return Stack Overflow: Mitigation: Safe RET718worker1 # [ 0.228295] Speculative Return Stack Overflow: Mitigation: Safe RET719client # [ 0.225781] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256720client # [ 0.226781] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64721client # [ 0.227780] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512722client # [ 0.228781] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024723lb # [ 0.213223] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization724worker1 # [ 0.229293] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization725client # [ 0.229780] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8726lb # [ 0.215230] Spectre V2 : Enabling IBPB for BPF727worker1 # [ 0.230588] Spectre V2 : Enabling IBPB for BPF728client # [ 0.230780] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.729lb # [ 0.216224] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier730worker1 # [ 0.231581] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier731lb # [ 0.217224] active return thunk: srso_alias_return_thunk732worker1 # [ 0.232580] active return thunk: srso_alias_return_thunk733lb # [ 0.218244] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'734worker1 # [ 0.234268] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'735lb # [ 0.219224] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'736worker1 # [ 0.235518] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'737lb # [ 0.220223] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'738worker1 # [ 0.236579] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'739lb # [ 0.221223] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'740worker1 # [ 0.237579] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'741lb # [ 0.222223] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'742worker1 # [ 0.238580] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'743lb # [ 0.223223] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'744worker1 # [ 0.239579] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'745lb # [ 0.224223] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'746worker1 # [ 0.240580] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'747lb # [ 0.226223] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256748worker1 # [ 0.241580] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256749lb # [ 0.227223] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64750worker1 # [ 0.242580] x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64751lb # [ 0.228223] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512752worker1 # [ 0.243579] x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512753lb # [ 0.229223] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024754worker1 # [ 0.244580] x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024755lb # [ 0.230176] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8756worker1 # [ 0.245580] x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8757lb # [ 0.230975] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.758worker1 # [ 0.246580] x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using 'compacted' format.759worker2 # [ 0.272122] Freeing SMP alternatives memory: 44K760worker2 # [ 0.272568] pid_max: default: 32768 minimum: 301761worker2 # [ 0.273651] LSM: initializing lsm=capability,landlock,yama,bpf,ima762worker2 # [ 0.274696] landlock: Up and running.763worker2 # [ 0.275566] Yama: becoming mindful.764worker2 # [ 0.276398] LSM support for eBPF active765worker2 # [ 0.277182] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)766client # [ 0.264826] Freeing SMP alternatives memory: 44K767worker2 # [ 0.278596] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)768client # [ 0.265603] pid_max: default: 32768 minimum: 301769client # [ 0.266436] LSM: initializing lsm=capability,landlock,yama,bpf,ima770worker2 # [ 0.280881] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)771client # [ 0.266887] landlock: Up and running.772client # [ 0.267781] Yama: becoming mindful.773worker2 # [ 0.282095] Performance Events: Fam17h+ core perfctr, AMD PMU driver.774client # [ 0.269002] LSM support for eBPF active775worker2 # [ 0.282575] ... version: 2776client # [ 0.269832] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)777worker2 # [ 0.283325] ... bit width: 48778worker2 # [ 0.283568] ... generic counters: 6779client # [ 0.270819] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)780worker2 # [ 0.284297] ... generic bitmap: 000000000000003f781worker2 # [ 0.284568] ... fixed-purpose counters: 0782worker2 # [ 0.285323] ... fixed-purpose bitmap: 0000000000000000783client # [ 0.273281] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)784worker2 # [ 0.285568] ... value mask: 0000ffffffffffff785worker2 # [ 0.286471] ... max period: 00007fffffffffff786client # [ 0.274305] Performance Events: Fam17h+ core perfctr, AMD PMU driver.787client # [ 0.274785] ... version: 2788worker2 # [ 0.287235] ... global_ctrl mask: 000000000000003f789client # [ 0.275507] ... bit width: 48790worker2 # [ 0.287664] signal: max sigframe size: 3376791client # [ 0.275783] ... generic counters: 6792worker2 # [ 0.288475] rcu: Hierarchical SRCU implementation.793client # [ 0.276504] ... generic bitmap: 000000000000003f794worker2 # [ 0.289164] rcu: Max phase no-delay instances is 400.795client # [ 0.276794] ... fixed-purpose counters: 0796lb # [ 0.265519] Freeing SMP alternatives memory: 44K797client # [ 0.277514] ... fixed-purpose bitmap: 0000000000000000798lb # [ 0.266226] pid_max: default: 32768 minimum: 301799client # [ 0.277783] ... value mask: 0000ffffffffffff800worker1 # [ 0.281083] Freeing SMP alternatives memory: 44K801worker1 # [ 0.281584] pid_max: default: 32768 minimum: 301802lb # [ 0.267049] LSM: initializing lsm=capability,landlock,yama,bpf,ima803client # [ 0.278666] ... max period: 00007fffffffffff804lb # [ 0.268241] landlock: Up and running.805client # [ 0.279418] ... global_ctrl mask: 000000000000003f806worker1 # [ 0.282682] LSM: initializing lsm=capability,landlock,yama,bpf,ima807lb # [ 0.268845] Yama: becoming mindful.808client # [ 0.279892] signal: max sigframe size: 3376809worker1 # [ 0.283692] landlock: Up and running.810lb # [ 0.269435] LSM support for eBPF active811worker1 # [ 0.284315] Yama: becoming mindful.812worker2 # [ 0.294297] smp: Bringing up secondary CPUs ...813client # [ 0.280695] rcu: Hierarchical SRCU implementation.814worker2 # [ 0.294582] smp: Brought up 1 node, 1 CPU815worker1 # [ 0.285227] LSM support for eBPF active816client # [ 0.281379] rcu: Max phase no-delay instances is 400.817lb # [ 0.270332] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)818worker2 # [ 0.295259] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)819worker1 # [ 0.285709] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)820lb # [ 0.271266] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)821worker1 # [ 0.286622] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)822worker2 # [ 0.295748] Memory: 1454568K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)823lb # [ 0.273902] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)824worker2 # [ 0.296796] devtmpfs: initialized825worker2 # [ 0.297627] x86/mm: Memory block size: 128MB826client # [ 0.286242] smp: Bringing up secondary CPUs ...827worker1 # [ 0.289964] smpboot: CPU0: AMD EPYC 9654 96-Core Processor (family: 0x19, model: 0x11, stepping: 0x1)828lb # [ 0.274764] Performance Events: Fam17h+ core perfctr, AMD PMU driver.829client # [ 0.286797] smp: Brought up 1 node, 1 CPU830lb # [ 0.275228] ... version: 2831lb # [ 0.275951] ... bit width: 48832client # [ 0.287492] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)833worker1 # [ 0.291193] Performance Events: Fam17h+ core perfctr, AMD PMU driver.834worker2 # [ 0.299439] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)835lb # [ 0.276226] ... generic counters: 6836worker1 # [ 0.291589] ... version: 2837lb # [ 0.276900] ... generic bitmap: 000000000000003f838worker1 # [ 0.292364] ... bit width: 48839worker2 # [ 0.300483] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).840lb # [ 0.277235] ... fixed-purpose counters: 0841client # [ 0.287931] Memory: 1454548K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)842worker1 # [ 0.292582] ... generic counters: 6843worker2 # [ 0.301653] pinctrl core: initialized pinctrl subsystem844lb # [ 0.277931] ... fixed-purpose bitmap: 0000000000000000845client # [ 0.288998] devtmpfs: initialized846worker1 # [ 0.293303] ... generic bitmap: 000000000000003f847worker2 # [ 0.302787] PM: RTC time: 12:57:28, date: 2026-09-21848lb # [ 0.278225] ... value mask: 0000ffffffffffff849client # [ 0.289864] x86/mm: Memory block size: 128MB850worker1 # [ 0.293582] ... fixed-purpose counters: 0851lb # [ 0.279109] ... max period: 00007fffffffffff852worker1 # [ 0.294323] ... fixed-purpose bitmap: 0000000000000000853lb # [ 0.279921] ... global_ctrl mask: 000000000000003f854worker1 # [ 0.294582] ... value mask: 0000ffffffffffff855client # [ 0.291579] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)856lb # [ 0.280353] signal: max sigframe size: 3376857worker1 # [ 0.295556] ... max period: 00007fffffffffff858lb # [ 0.281167] rcu: Hierarchical SRCU implementation.859worker2 # [ 0.306339] NET: Registered PF_NETLINK/PF_ROUTE protocol family860client # [ 0.292691] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).861worker1 # [ 0.296337] ... global_ctrl mask: 000000000000003f862lb # [ 0.281764] rcu: Max phase no-delay instances is 400.863client # [ 0.293862] pinctrl core: initialized pinctrl subsystem864worker1 # [ 0.296694] signal: max sigframe size: 3376865worker2 # [ 0.307780] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations866client # [ 0.294984] PM: RTC time: 12:57:28, date: 2026-09-21867worker1 # [ 0.297594] rcu: Hierarchical SRCU implementation.868worker1 # [ 0.298427] rcu: Max phase no-delay instances is 400.869worker2 # [ 0.308591] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations870worker2 # [ 0.309843] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations871worker2 # [ 0.310578] audit: initializing netlink subsys (disabled)872client # [ 0.298438] NET: Registered PF_NETLINK/PF_ROUTE protocol family873lb # [ 0.286783] smp: Bringing up secondary CPUs ...874lb # [ 0.287239] smp: Brought up 1 node, 1 CPU875worker2 # [ 0.311701] thermal_sys: Registered thermal governor 'fair_share'876client # [ 0.299278] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations877lb # [ 0.287939] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)878worker2 # [ 0.311703] thermal_sys: Registered thermal governor 'bang_bang'879worker1 # [ 0.303889] smp: Bringing up secondary CPUs ...880client # [ 0.299808] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations881worker2 # [ 0.312569] thermal_sys: Registered thermal governor 'step_wise'882worker1 # [ 0.304597] smp: Brought up 1 node, 1 CPU883worker2 # [ 0.313512] thermal_sys: Registered thermal governor 'user_space'884lb # [ 0.288438] Memory: 1454592K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109348K reserved, 0K cma-reserved)885client # [ 0.301048] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations886worker1 # [ 0.305329] smpboot: Total of 1 processors activated (4800.02 BogoMIPS)887lb # [ 0.289435] devtmpfs: initialized888client # [ 0.301793] audit: initializing netlink subsys (disabled)889worker2 # [ 0.314337] audit: type=2000 audit(1789995449.163:1): state=initialized audit_enabled=0 res=1890lb # [ 0.290178] x86/mm: Memory block size: 128MB891worker2 # [ 0.316571] thermal_sys: Registered thermal governor 'power_allocator'892client # [ 0.302949] thermal_sys: Registered thermal governor 'fair_share'893worker1 # [ 0.305739] Memory: 1454548K/1572312K available (17223K kernel code, 2726K rwdata, 13584K rodata, 3644K init, 2988K bss, 109344K reserved, 0K cma-reserved)894worker2 # [ 0.316585] cpuidle: using governor menu895client # [ 0.302951] thermal_sys: Registered thermal governor 'bang_bang'896worker1 # [ 0.306849] devtmpfs: initialized897lb # [ 0.291748] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)898client # [ 0.303784] thermal_sys: Registered thermal governor 'step_wise'899worker1 # [ 0.307616] x86/mm: Memory block size: 128MB900worker2 # [ 0.318810] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5901client # [ 0.304710] thermal_sys: Registered thermal governor 'user_space'902lb # [ 0.292250] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).903worker2 # [ 0.319792] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]904lb # [ 0.293319] pinctrl core: initialized pinctrl subsystem905client # [ 0.305503] audit: type=2000 audit(1789995449.172:1): state=initialized audit_enabled=0 res=1906worker1 # [ 0.309407] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)907worker2 # [ 0.320571] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry908lb # [ 0.294402] PM: RTC time: 12:57:28, date: 2026-09-21909client # [ 0.306786] thermal_sys: Registered thermal governor 'power_allocator'910worker2 # [ 0.321581] PCI: Using configuration type 1 for base access911client # [ 0.306809] cpuidle: using governor menu912worker1 # [ 0.310488] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).913worker1 # [ 0.311670] pinctrl core: initialized pinctrl subsystem914client # [ 0.309004] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5915worker2 # [ 0.322722] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.916worker1 # [ 0.312892] PM: RTC time: 12:57:28, date: 2026-09-21917lb # [ 0.297863] NET: Registered PF_NETLINK/PF_ROUTE protocol family918client # [ 0.310005] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]919client # [ 0.310786] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry920lb # [ 0.298735] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations921client # [ 0.311795] PCI: Using configuration type 1 for base access922lb # [ 0.299263] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations923worker1 # [ 0.316370] NET: Registered PF_NETLINK/PF_ROUTE protocol family924client # [ 0.312915] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.925worker2 # [ 0.327648] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages926lb # [ 0.300493] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations927worker1 # [ 0.317862] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations928worker2 # [ 0.328569] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page929lb # [ 0.301236] audit: initializing netlink subsys (disabled)930worker1 # [ 0.318607] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations931lb # [ 0.302341] thermal_sys: Registered thermal governor 'fair_share'932lb # [ 0.302343] thermal_sys: Registered thermal governor 'bang_bang'933worker1 # [ 0.319874] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations934lb # [ 0.303227] thermal_sys: Registered thermal governor 'step_wise'935worker1 # [ 0.320594] audit: initializing netlink subsys (disabled)936lb # [ 0.304208] thermal_sys: Registered thermal governor 'user_space'937worker2 # [ 0.333569] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages938worker1 # [ 0.321759] thermal_sys: Registered thermal governor 'fair_share'939lb # [ 0.304989] audit: type=2000 audit(1789995449.187:1): state=initialized audit_enabled=0 res=1940worker2 # [ 0.334569] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page941worker1 # [ 0.321762] thermal_sys: Registered thermal governor 'bang_bang'942client # [ 0.320046] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages943lb # [ 0.306228] thermal_sys: Registered thermal governor 'power_allocator'944worker1 # [ 0.322583] thermal_sys: Registered thermal governor 'step_wise'945client # [ 0.320784] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page946lb # [ 0.306251] cpuidle: using governor menu947worker1 # [ 0.323557] thermal_sys: Registered thermal governor 'user_space'948lb # [ 0.308438] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5949worker1 # [ 0.324326] audit: type=2000 audit(1789995449.183:1): state=initialized audit_enabled=0 res=1950lb # [ 0.309458] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]951worker1 # [ 0.325585] thermal_sys: Registered thermal governor 'power_allocator'952worker1 # [ 0.325599] cpuidle: using governor menu953lb # [ 0.310229] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry954client # [ 0.325784] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages955lb # [ 0.311237] PCI: Using configuration type 1 for base access956worker1 # [ 0.327821] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5957client # [ 0.326784] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page958lb # [ 0.312326] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.959worker1 # [ 0.328831] PCI: ECAM [mem 0xb0000000-0xbfffffff] (base 0xb0000000) for domain 0000 [bus 00-ff]960worker1 # [ 0.329585] PCI: ECAM [mem 0xb0000000-0xbfffffff] reserved as E820 entry961worker2 # [ 0.344934] ACPI: Added _OSI(Module Device)962worker1 # [ 0.330595] PCI: Using configuration type 1 for base access963worker2 # [ 0.345570] ACPI: Added _OSI(Processor Device)964worker2 # [ 0.346352] ACPI: Added _OSI(Processor Aggregator Device)965worker1 # [ 0.331753] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.966client # [ 0.335323] ACPI: Added _OSI(Module Device)967lb # [ 0.319457] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages968client # [ 0.335785] ACPI: Added _OSI(Processor Device)969lb # [ 0.320227] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page970client # [ 0.336546] ACPI: Added _OSI(Processor Aggregator Device)971worker1 # [ 0.336874] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages972worker1 # [ 0.337584] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page973worker2 # [ 0.354804] ACPI: 1 ACPI AML tables successfully acquired and loaded974lb # [ 0.325227] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages975lb # [ 0.326227] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page976worker1 # [ 0.342584] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages977client # [ 0.343315] ACPI: 1 ACPI AML tables successfully acquired and loaded978worker1 # [ 0.343582] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page979worker2 # [ 0.358975] ACPI: Interpreter enabled980worker2 # [ 0.359582] ACPI: PM: (supports S0 S3 S4 S5)981worker2 # [ 0.360334] ACPI: Using IOAPIC for interrupt routing982client # [ 0.346728] ACPI: Interpreter enabled983client # [ 0.347281] ACPI: PM: (supports S0 S3 S4 S5)984client # [ 0.347784] ACPI: Using IOAPIC for interrupt routing985worker2 # [ 0.362603] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug986lb # [ 0.334843] ACPI: Added _OSI(Module Device)987worker2 # [ 0.363569] PCI: Using E820 reservations for host bridge windows988client # [ 0.348636] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug989lb # [ 0.335228] ACPI: Added _OSI(Processor Device)990lb # [ 0.335981] ACPI: Added _OSI(Processor Aggregator Device)991worker2 # [ 0.366645] ACPI: Enabled 2 GPEs in block 00 to 3F992worker1 # [ 0.353988] ACPI: Added _OSI(Module Device)993worker1 # [ 0.354584] ACPI: Added _OSI(Processor Device)994client # [ 0.353784] PCI: Using E820 reservations for host bridge windows995worker1 # [ 0.355328] ACPI: Added _OSI(Processor Aggregator Device)996client # [ 0.354881] ACPI: Enabled 2 GPEs in block 00 to 3F997lb # [ 0.342787] ACPI: 1 ACPI AML tables successfully acquired and loaded998worker2 # [ 0.374172] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])999worker2 # [ 0.374575] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1000lb # [ 0.346144] ACPI: Interpreter enabled1001lb # [ 0.346700] ACPI: PM: (supports S0 S3 S4 S5)1002worker2 # [ 0.375647] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1003lb # [ 0.347226] ACPI: Using IOAPIC for interrupt routing1004worker2 # [ 0.376690] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1005client # [ 0.363504] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1006worker1 # [ 0.363857] ACPI: 1 ACPI AML tables successfully acquired and loaded1007worker2 # [ 0.377922] PCI host bridge to bus 0000:001008lb # [ 0.348134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1009client # [ 0.364376] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1010worker2 # [ 0.378573] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1011worker2 # [ 0.379569] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1012client # [ 0.364861] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1013worker1 # [ 0.368000] ACPI: Interpreter enabled1014worker2 # [ 0.380568] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1015client # [ 0.365906] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1016worker1 # [ 0.368598] ACPI: PM: (supports S0 S3 S4 S5)1017client # [ 0.367158] PCI host bridge to bus 0000:001018worker2 # [ 0.381569] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1019lb # [ 0.353226] PCI: Using E820 reservations for host bridge windows1020worker1 # [ 0.369395] ACPI: Using IOAPIC for interrupt routing1021client # [ 0.367788] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1022lb # [ 0.354313] ACPI: Enabled 2 GPEs in block 00 to 3F1023worker2 # [ 0.382569] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1024client # [ 0.368784] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1025worker2 # [ 0.383569] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1026client # [ 0.369783] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1027worker2 # [ 0.384569] pci_bus 0000:00: root bus resource [bus 00-ff]1028worker1 # [ 0.372658] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug1029client # [ 0.370784] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1030worker2 # [ 0.385505] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1031worker1 # [ 0.373590] PCI: Using E820 reservations for host bridge windows1032client # [ 0.371783] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1033worker2 # [ 0.387038] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1034client # [ 0.372784] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1035client # [ 0.373784] pci_bus 0000:00: root bus resource [bus 00-ff]1036worker1 # [ 0.376723] ACPI: Enabled 2 GPEs in block 00 to 3F1037client # [ 0.374733] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1038lb # [ 0.362938] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1039client # [ 0.376250] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1040lb # [ 0.363782] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1041lb # [ 0.364320] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1042lb # [ 0.365350] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1043worker2 # [ 0.389688] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1044lb # [ 0.366555] PCI host bridge to bus 0000:001045worker1 # [ 0.383984] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])1046worker2 # [ 0.390624] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1047lb # [ 0.367220] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1048worker2 # [ 0.391590] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1049worker1 # [ 0.384590] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]1050lb # [ 0.368226] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1051worker1 # [ 0.385660] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR]1052worker2 # [ 0.392900] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1053lb # [ 0.369226] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1054client # [ 0.378888] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1055worker1 # [ 0.386704] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability]1056lb # [ 0.370226] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1057client # [ 0.379795] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1058worker2 # [ 0.394313] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1059worker1 # [ 0.387998] PCI host bridge to bus 0000:001060client # [ 0.380751] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1061lb # [ 0.371225] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1062worker1 # [ 0.388587] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]1063lb # [ 0.372226] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1064client # [ 0.381736] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1065worker1 # [ 0.389583] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]1066lb # [ 0.373226] pci_bus 0000:00: root bus resource [bus 00-ff]1067worker2 # [ 0.396596] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1068worker1 # [ 0.390583] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]1069lb # [ 0.374159] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1070worker2 # [ 0.397497] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1071client # [ 0.383501] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1072worker1 # [ 0.391583] pci_bus 0000:00: root bus resource [mem 0x60000000-0xafffffff window]1073worker2 # [ 0.398325] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1074lb # [ 0.375659] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1075worker2 # [ 0.399434] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1076worker1 # [ 0.392583] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]1077client # [ 0.386819] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1078worker1 # [ 0.393583] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window]1079client # [ 0.387677] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1080worker2 # [ 0.400954] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1081worker1 # [ 0.394583] pci_bus 0000:00: root bus resource [bus 00-ff]1082client # [ 0.388529] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1083worker1 # [ 0.395624] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000 conventional PCI endpoint1084client # [ 0.388790] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1085worker2 # [ 0.403597] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1086worker1 # [ 0.397089] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint1087worker2 # [ 0.404525] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1088client # [ 0.390873] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1089worker2 # [ 0.405332] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1090lb # [ 0.378291] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1091worker2 # [ 0.406971] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1092client # [ 0.392790] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1093lb # [ 0.379239] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1094client # [ 0.393677] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1095lb # [ 0.380201] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1096client # [ 0.394553] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1097lb # [ 0.381389] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1098worker2 # [ 0.409794] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1099client # [ 0.396243] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1100worker2 # [ 0.411260] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1101lb # [ 0.382949] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1102worker1 # [ 0.399727] pci 0000:00:01.0: BAR 0 [mem 0xfd000000-0xfdffffff pref]1103worker2 # [ 0.411592] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1104worker1 # [ 0.400596] pci 0000:00:01.0: BAR 2 [mem 0xfebd0000-0xfebd0fff]1105lb # [ 0.385252] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1106worker2 # [ 0.413179] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1107worker1 # [ 0.401609] pci 0000:00:01.0: ROM [mem 0xfebc0000-0xfebcffff pref]1108client # [ 0.399066] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1109lb # [ 0.386103] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1110client # [ 0.400421] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1111worker1 # [ 0.402954] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]1112lb # [ 0.386977] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1113client # [ 0.400805] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1114worker2 # [ 0.414598] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1115lb # [ 0.387232] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1116worker2 # [ 0.415445] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1117worker1 # [ 0.404352] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1118client # [ 0.402330] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1119worker2 # [ 0.416361] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1120lb # [ 0.389066] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1121client # [ 0.403818] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1122worker2 # [ 0.418653] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1123worker1 # [ 0.407617] pci 0000:00:02.0: BAR 0 [io 0xc140-0xc15f]1124lb # [ 0.391251] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1125client # [ 0.404678] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1126worker1 # [ 0.408478] pci 0000:00:02.0: BAR 1 [mem 0xfebd1000-0xfebd1fff]1127lb # [ 0.392055] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1128client # [ 0.405512] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1129lb # [ 0.392921] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1130worker1 # [ 0.409315] pci 0000:00:02.0: BAR 4 [mem 0xfe000000-0xfe003fff 64bit pref]1131worker2 # [ 0.420601] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1132worker2 # [ 0.421496] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1133worker1 # [ 0.409613] pci 0000:00:02.0: ROM [mem 0xfeb40000-0xfeb7ffff pref]1134client # [ 0.406319] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1135lb # [ 0.393812] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1136worker2 # [ 0.422342] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1137worker1 # [ 0.411202] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1138worker2 # [ 0.424031] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1139client # [ 0.408814] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1140client # [ 0.409678] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1141lb # [ 0.396405] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1142client # [ 0.410534] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1143worker1 # [ 0.413614] pci 0000:00:03.0: BAR 0 [io 0xc160-0xc17f]1144lb # [ 0.397232] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1145worker1 # [ 0.414438] pci 0000:00:03.0: BAR 1 [mem 0xfebd2000-0xfebd2fff]1146lb # [ 0.398216] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1147worker2 # [ 0.426848] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1148client # [ 0.411399] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1149worker1 # [ 0.415389] pci 0000:00:03.0: BAR 4 [mem 0xfe004000-0xfe007fff 64bit pref]1150worker2 # [ 0.428211] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1151lb # [ 0.399608] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1152worker2 # [ 0.428594] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1153worker1 # [ 0.417151] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1154lb # [ 0.401233] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1155client # [ 0.413762] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1156worker2 # [ 0.430222] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1157client # [ 0.414790] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1158lb # [ 0.402076] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1159lb # [ 0.402977] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1160client # [ 0.415742] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1161worker1 # [ 0.419544] pci 0000:00:04.0: BAR 0 [io 0xc080-0xc0bf]1162worker2 # [ 0.431576] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1163worker1 # [ 0.420275] pci 0000:00:04.0: BAR 1 [mem 0xfebd3000-0xfebd3fff]1164worker2 # [ 0.432432] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1165client # [ 0.417167] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1166lb # [ 0.403785] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1167worker2 # [ 0.433345] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1168worker1 # [ 0.420609] pci 0000:00:04.0: BAR 4 [mem 0xfe008000-0xfe00bfff 64bit pref]1169worker2 # [ 0.434405] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1170client # [ 0.418818] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1171lb # [ 0.405251] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1172worker1 # [ 0.422180] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1173client # [ 0.419670] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1174lb # [ 0.406078] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1175worker2 # [ 0.436154] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1176client # [ 0.420526] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1177lb # [ 0.406995] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1178client # [ 0.420789] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1179worker2 # [ 0.437558] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1180lb # [ 0.407816] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1181worker1 # [ 0.424821] pci 0000:00:05.0: BAR 0 [io 0xc180-0xc19f]1182worker2 # [ 0.438320] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1183client # [ 0.422320] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1184worker1 # [ 0.426270] pci 0000:00:05.0: BAR 1 [mem 0xfebd4000-0xfebd4fff]1185worker2 # [ 0.439129] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1186lb # [ 0.410191] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1187worker1 # [ 0.426608] pci 0000:00:05.0: BAR 4 [mem 0xfe00c000-0xfe00ffff 64bit pref]1188client # [ 0.424790] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1189lb # [ 0.410844] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1190client # [ 0.425748] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1191worker1 # [ 0.428134] pci 0000:00:06.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1192worker2 # [ 0.440607] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1193lb # [ 0.411248] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1194client # [ 0.427175] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1195worker2 # [ 0.441508] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1196lb # [ 0.412770] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1197worker1 # [ 0.430441] pci 0000:00:06.0: BAR 0 [io 0xc1a0-0xc1bf]1198worker2 # [ 0.442593] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1199worker1 # [ 0.431302] pci 0000:00:06.0: BAR 1 [mem 0xfebd5000-0xfebd5fff]1200client # [ 0.428827] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1201lb # [ 0.414234] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1202client # [ 0.429686] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1203worker1 # [ 0.431608] pci 0000:00:06.0: BAR 4 [mem 0xfe010000-0xfe013fff 64bit pref]1204worker2 # [ 0.444161] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1205lb # [ 0.415127] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1206lb # [ 0.415936] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1207client # [ 0.430499] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1208lb # [ 0.416232] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1209worker2 # [ 0.445599] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1210worker1 # [ 0.433893] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1211client # [ 0.431356] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1212worker2 # [ 0.446575] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1213lb # [ 0.417771] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1214worker2 # [ 0.447530] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1215client # [ 0.432818] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1216worker1 # [ 0.436558] pci 0000:00:07.0: BAR 0 [io 0xc000-0xc07f]1217lb # [ 0.419018] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1218client # [ 0.433677] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1219worker1 # [ 0.437330] pci 0000:00:07.0: BAR 1 [mem 0xfebd6000-0xfebd6fff]1220worker2 # [ 0.449077] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1221lb # [ 0.419947] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1222client # [ 0.434473] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1223worker1 # [ 0.437605] pci 0000:00:07.0: BAR 4 [mem 0xfe014000-0xfe017fff 64bit pref]1224worker2 # [ 0.450273] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1225lb # [ 0.420780] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1226client # [ 0.435375] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1227worker1 # [ 0.439216] pci 0000:00:08.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1228worker2 # [ 0.450804] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1229lb # [ 0.422234] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1230worker2 # [ 0.452226] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1231client # [ 0.436687] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1232lb # [ 0.423087] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1233lb # [ 0.423986] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1234worker1 # [ 0.441616] pci 0000:00:08.0: BAR 0 [io 0xc1c0-0xc1df]1235client # [ 0.438119] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1236worker1 # [ 0.442550] pci 0000:00:08.0: BAR 1 [mem 0xfebd7000-0xfebd7fff]1237client # [ 0.439596] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1238worker1 # [ 0.443429] pci 0000:00:08.0: BAR 4 [mem 0xfe018000-0xfe01bfff 64bit pref]1239lb # [ 0.424795] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1240worker1 # [ 0.444522] pci 0000:00:08.0: ROM [mem 0xfeb80000-0xfebbffff pref]1241lb # [ 0.427251] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1242lb # [ 0.428153] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1243worker1 # [ 0.446043] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1244lb # [ 0.428973] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1245worker1 # [ 0.447598] pci 0000:00:09.0: BAR 1 [mem 0xfebd8000-0xfebd8fff]1246lb # [ 0.429784] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1247worker1 # [ 0.448609] pci 0000:00:09.0: BAR 4 [mem 0xfe01c000-0xfe01ffff 64bit pref]1248lb # [ 0.431011] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1249worker1 # [ 0.450222] pci 0000:00:0a.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1250lb # [ 0.432080] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1251worker2 # [ 0.452784] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1252lb # [ 0.433779] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1253worker2 # [ 0.455231] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1254worker1 # [ 0.452386] pci 0000:00:0a.0: BAR 0 [io 0xc0c0-0xc0ff]1255client # [ 0.440584] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1256worker2 # [ 0.455833] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1257worker1 # [ 0.453302] pci 0000:00:0a.0: BAR 1 [mem 0xfebd9000-0xfebd9fff]1258client # [ 0.442486] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1259worker1 # [ 0.453606] pci 0000:00:0a.0: BAR 4 [mem 0xfe020000-0xfe023fff 64bit pref]1260worker2 # [ 0.457202] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1261client # [ 0.443052] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1262worker1 # [ 0.455198] pci 0000:00:0b.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1263worker2 # [ 0.457847] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1264client # [ 0.444416] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1265worker2 # [ 0.458857] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1266client # [ 0.445033] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1267worker2 # [ 0.459840] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1268worker1 # [ 0.457616] pci 0000:00:0b.0: BAR 0 [io 0xc1e0-0xc1ff]1269client # [ 0.446070] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1270worker1 # [ 0.458533] pci 0000:00:0b.0: BAR 1 [mem 0xfebda000-0xfebdafff]1271worker2 # [ 0.461598] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1272worker1 # [ 0.459394] pci 0000:00:0b.0: BAR 4 [mem 0xfe024000-0xfe027fff 64bit pref]1273client # [ 0.447085] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1274worker2 # [ 0.462523] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1275lb # [ 0.434420] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1276worker1 # [ 0.461097] pci 0000:00:1d.0: [8086:2934] type 00 class 0x0c0300 conventional PCI endpoint1277client # [ 0.448811] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1278worker2 # [ 0.463932] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1279lb # [ 0.436797] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1280client # [ 0.449664] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1281worker1 # [ 0.462348] pci 0000:00:1d.0: BAR 4 [io 0xc200-0xc21f]1282worker2 # [ 0.465231] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1283lb # [ 0.437506] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1284client # [ 0.450856] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1285worker1 # [ 0.462801] pci 0000:00:1d.1: [8086:2935] type 00 class 0x0c0300 conventional PCI endpoint1286lb # [ 0.438742] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1287client # [ 0.452789] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1288lb # [ 0.439443] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1289worker1 # [ 0.464643] pci 0000:00:1d.1: BAR 4 [io 0xc220-0xc23f]1290worker2 # [ 0.468577] ACPI: PCI: Interrupt link LNKA configured for IRQ 101291lb # [ 0.440857] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1292worker2 # [ 0.469680] ACPI: PCI: Interrupt link LNKB configured for IRQ 101293client # [ 0.456571] ACPI: PCI: Interrupt link LNKA configured for IRQ 101294worker2 # [ 0.470671] ACPI: PCI: Interrupt link LNKC configured for IRQ 111295lb # [ 0.441470] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1296client # [ 0.457594] ACPI: PCI: Interrupt link LNKB configured for IRQ 101297worker2 # [ 0.471668] ACPI: PCI: Interrupt link LNKD configured for IRQ 111298client # [ 0.458586] ACPI: PCI: Interrupt link LNKC configured for IRQ 111299lb # [ 0.443027] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1300worker2 # [ 0.472669] ACPI: PCI: Interrupt link LNKE configured for IRQ 101301client # [ 0.459618] ACPI: PCI: Interrupt link LNKD configured for IRQ 111302lb # [ 0.444166] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1303worker2 # [ 0.473680] ACPI: PCI: Interrupt link LNKF configured for IRQ 101304client # [ 0.460622] ACPI: PCI: Interrupt link LNKE configured for IRQ 101305worker2 # [ 0.474670] ACPI: PCI: Interrupt link LNKG configured for IRQ 111306lb # [ 0.445282] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1307client # [ 0.461630] ACPI: PCI: Interrupt link LNKF configured for IRQ 101308worker2 # [ 0.475667] ACPI: PCI: Interrupt link LNKH configured for IRQ 111309client # [ 0.462737] ACPI: PCI: Interrupt link LNKG configured for IRQ 111310lb # [ 0.446794] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1311worker2 # [ 0.476608] ACPI: PCI: Interrupt link GSIA configured for IRQ 161312client # [ 0.463616] ACPI: PCI: Interrupt link LNKH configured for IRQ 111313worker2 # [ 0.477583] ACPI: PCI: Interrupt link GSIB configured for IRQ 171314client # [ 0.464563] ACPI: PCI: Interrupt link GSIA configured for IRQ 161315worker2 # [ 0.478584] ACPI: PCI: Interrupt link GSIC configured for IRQ 181316client # [ 0.464800] ACPI: PCI: Interrupt link GSIB configured for IRQ 171317worker2 # [ 0.479580] ACPI: PCI: Interrupt link GSID configured for IRQ 191318worker1 # [ 0.465777] pci 0000:00:1d.2: [8086:2936] type 00 class 0x0c0300 conventional PCI endpoint1319client # [ 0.465759] ACPI: PCI: Interrupt link GSIC configured for IRQ 181320lb # [ 0.450520] ACPI: PCI: Interrupt link LNKA configured for IRQ 101321worker2 # [ 0.480576] ACPI: PCI: Interrupt link GSIE configured for IRQ 201322client # [ 0.466511] ACPI: PCI: Interrupt link GSID configured for IRQ 191323lb # [ 0.451336] ACPI: PCI: Interrupt link LNKB configured for IRQ 101324worker1 # [ 0.468408] pci 0000:00:1d.2: BAR 4 [io 0xc240-0xc25f]1325worker2 # [ 0.481584] ACPI: PCI: Interrupt link GSIF configured for IRQ 211326client # [ 0.466798] ACPI: PCI: Interrupt link GSIE configured for IRQ 201327lb # [ 0.452326] ACPI: PCI: Interrupt link LNKC configured for IRQ 111328worker2 # [ 0.482579] ACPI: PCI: Interrupt link GSIG configured for IRQ 221329client # [ 0.467742] ACPI: PCI: Interrupt link GSIF configured for IRQ 211330worker1 # [ 0.469541] pci 0000:00:1d.7: [8086:293a] type 00 class 0x0c0320 conventional PCI endpoint1331lb # [ 0.453326] ACPI: PCI: Interrupt link LNKD configured for IRQ 111332worker2 # [ 0.483571] ACPI: PCI: Interrupt link GSIH configured for IRQ 231333client # [ 0.468513] ACPI: PCI: Interrupt link GSIG configured for IRQ 221334lb # [ 0.454322] ACPI: PCI: Interrupt link LNKE configured for IRQ 101335client # [ 0.468795] ACPI: PCI: Interrupt link GSIH configured for IRQ 231336worker1 # [ 0.471310] pci 0000:00:1d.7: BAR 0 [mem 0xfebdb000-0xfebdbfff]1337lb # [ 0.455322] ACPI: PCI: Interrupt link LNKF configured for IRQ 101338worker2 # [ 0.485633] iommu: Default domain type: Translated1339lb # [ 0.456323] ACPI: PCI: Interrupt link LNKG configured for IRQ 111340worker2 # [ 0.486456] iommu: DMA domain TLB invalidation policy: lazy mode1341client # [ 0.470812] iommu: Default domain type: Translated1342worker1 # [ 0.471858] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100 conventional PCI endpoint1343client # [ 0.471561] iommu: DMA domain TLB invalidation policy: lazy mode1344worker2 # [ 0.487584] ACPI: bus type USB registered1345lb # [ 0.457321] ACPI: PCI: Interrupt link LNKH configured for IRQ 111346client # [ 0.472049] ACPI: bus type USB registered1347worker2 # [ 0.488384] usbcore: registered new interface driver usbfs1348worker1 # [ 0.472899] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO1349lb # [ 0.458280] ACPI: PCI: Interrupt link GSIA configured for IRQ 161350client # [ 0.472746] usbcore: registered new interface driver usbfs1351worker2 # [ 0.489264] usbcore: registered new interface driver hub1352lb # [ 0.459206] ACPI: PCI: Interrupt link GSIB configured for IRQ 171353client # [ 0.473434] usbcore: registered new interface driver hub1354worker1 # [ 0.473871] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601 conventional PCI endpoint1355worker2 # [ 0.489589] usbcore: registered new device driver usb1356lb # [ 0.459936] ACPI: PCI: Interrupt link GSIC configured for IRQ 181357client # [ 0.473795] usbcore: registered new device driver usb1358lb # [ 0.460242] ACPI: PCI: Interrupt link GSID configured for IRQ 191359worker2 # [ 0.491417] NetLabel: Initializing1360worker1 # [ 0.475635] pci 0000:00:1f.2: BAR 4 [io 0xc260-0xc27f]1361lb # [ 0.461212] ACPI: PCI: Interrupt link GSIE configured for IRQ 201362client # [ 0.475511] NetLabel: Initializing1363worker2 # [ 0.492023] NetLabel: domain hash size = 1281364client # [ 0.475789] NetLabel: domain hash size = 1281365lb # [ 0.461949] ACPI: PCI: Interrupt link GSIF configured for IRQ 211366worker1 # [ 0.476538] pci 0000:00:1f.2: BAR 5 [mem 0xfebdc000-0xfebdcfff]1367worker2 # [ 0.492573] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1368client # [ 0.476490] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1369lb # [ 0.462242] ACPI: PCI: Interrupt link GSIG configured for IRQ 221370worker2 # [ 0.493653] NetLabel: unlabeled traffic allowed by default1371worker1 # [ 0.477681] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500 conventional PCI endpoint1372client # [ 0.476837] NetLabel: unlabeled traffic allowed by default1373worker2 # [ 0.494578] PCI: Using ACPI for IRQ routing1374lb # [ 0.463195] ACPI: PCI: Interrupt link GSIH configured for IRQ 231375client # [ 0.477796] PCI: Using ACPI for IRQ routing1376lb # [ 0.464971] iommu: Default domain type: Translated1377worker1 # [ 0.479336] pci 0000:00:1f.3: BAR 4 [io 0x0700-0x073f]1378lb # [ 0.465233] iommu: DMA domain TLB invalidation policy: lazy mode1379lb # [ 0.466468] ACPI: bus type USB registered1380lb # [ 0.467265] usbcore: registered new interface driver usbfs1381lb # [ 0.468215] usbcore: registered new interface driver hub1382worker1 # [ 0.483006] ACPI: PCI: Interrupt link LNKA configured for IRQ 101383lb # [ 0.469014] usbcore: registered new device driver usb1384worker1 # [ 0.483699] ACPI: PCI: Interrupt link LNKB configured for IRQ 101385lb # [ 0.470165] NetLabel: Initializing1386worker1 # [ 0.484688] ACPI: PCI: Interrupt link LNKC configured for IRQ 111387lb # [ 0.470659] NetLabel: domain hash size = 1281388lb # [ 0.471226] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1389worker1 # [ 0.485684] ACPI: PCI: Interrupt link LNKD configured for IRQ 111390lb # [ 0.472190] NetLabel: unlabeled traffic allowed by default1391worker1 # [ 0.486683] ACPI: PCI: Interrupt link LNKE configured for IRQ 101392lb # [ 0.472951] PCI: Using ACPI for IRQ routing1393worker1 # [ 0.487691] ACPI: PCI: Interrupt link LNKF configured for IRQ 101394worker1 # [ 0.488696] ACPI: PCI: Interrupt link LNKG configured for IRQ 111395worker1 # [ 0.489695] ACPI: PCI: Interrupt link LNKH configured for IRQ 111396worker1 # [ 0.490619] ACPI: PCI: Interrupt link GSIA configured for IRQ 161397worker1 # [ 0.491610] ACPI: PCI: Interrupt link GSIB configured for IRQ 171398worker1 # [ 0.492599] ACPI: PCI: Interrupt link GSIC configured for IRQ 181399worker1 # [ 0.493595] ACPI: PCI: Interrupt link GSID configured for IRQ 191400worker1 # [ 0.494595] ACPI: PCI: Interrupt link GSIE configured for IRQ 201401worker1 # [ 0.495595] ACPI: PCI: Interrupt link GSIF configured for IRQ 211402worker1 # [ 0.496598] ACPI: PCI: Interrupt link GSIG configured for IRQ 221403worker1 # [ 0.497623] ACPI: PCI: Interrupt link GSIH configured for IRQ 231404worker1 # [ 0.499765] iommu: Default domain type: Translated1405worker1 # [ 0.500622] iommu: DMA domain TLB invalidation policy: lazy mode1406worker1 # [ 0.501872] ACPI: bus type USB registered1407worker1 # [ 0.502644] usbcore: registered new interface driver usbfs1408worker1 # [ 0.503599] usbcore: registered new interface driver hub1409worker1 # [ 0.504554] usbcore: registered new device driver usb1410worker1 # [ 0.506218] NetLabel: Initializing1411worker1 # [ 0.506583] NetLabel: domain hash size = 1281412worker1 # [ 0.507385] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO1413worker1 # [ 0.508400] NetLabel: unlabeled traffic allowed by default1414worker1 # [ 0.509344] PCI: Using ACPI for IRQ routing1415client # [ 0.522034] pci 0000:00:01.0: vgaarb: setting as boot VGA device1416client # [ 0.522779] pci 0000:00:01.0: vgaarb: bridge control possible1417client # [ 0.522779] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1418worker2 # [ 0.541033] pci 0000:00:01.0: vgaarb: setting as boot VGA device1419client # [ 0.522787] vgaarb: loaded1420worker2 # [ 0.541564] pci 0000:00:01.0: vgaarb: bridge control possible1421client # [ 0.523514] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01422client # [ 0.523785] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1423worker2 # [ 0.541564] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1424worker2 # [ 0.541573] vgaarb: loaded1425worker2 # [ 0.542367] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01426worker2 # [ 0.543208] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1427client # [ 0.526864] clocksource: Switched to clocksource kvm-clock1428lb # [ 0.516219] pci 0000:00:01.0: vgaarb: setting as boot VGA device1429lb # [ 0.516222] pci 0000:00:01.0: vgaarb: bridge control possible1430client # [ 0.530362] VFS: Disk quotas dquot_6.6.01431worker2 # [ 0.547665] clocksource: Switched to clocksource kvm-clock1432lb # [ 0.516222] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1433client # [ 0.531093] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1434lb # [ 0.516231] vgaarb: loaded1435client # [ 0.532440] pnp: PnP ACPI init1436worker2 # [ 0.549487] VFS: Disk quotas dquot_6.6.01437lb # [ 0.516975] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01438client # [ 0.533223] ACPI: IRQ 4 override to edge(!), high(!)1439lb # [ 0.517227] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1440client # [ 0.534221] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1441worker2 # [ 0.552153] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1442client # [ 0.535802] pnp: PnP ACPI: found 6 devices1443worker2 # [ 0.553628] pnp: PnP ACPI init1444lb # [ 0.519313] clocksource: Switched to clocksource kvm-clock1445worker2 # [ 0.554503] ACPI: IRQ 4 override to edge(!), high(!)1446worker2 # [ 0.555544] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1447lb # [ 0.522690] VFS: Disk quotas dquot_6.6.01448worker2 # [ 0.557207] pnp: PnP ACPI: found 6 devices1449lb # [ 0.523407] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1450lb # [ 0.524820] pnp: PnP ACPI init1451lb # [ 0.525602] ACPI: IRQ 4 override to edge(!), high(!)1452client # [ 0.543456] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1453lb # [ 0.526611] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1454client # [ 0.544895] clocksource: Switched to clocksource acpi_pm1455lb # [ 0.528149] pnp: PnP ACPI: found 6 devices1456client # [ 0.545872] NET: Registered PF_INET protocol family1457client # [ 0.547081] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1458worker2 # [ 0.564960] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1459worker2 # [ 0.566552] clocksource: Switched to clocksource acpi_pm1460worker2 # [ 0.567610] NET: Registered PF_INET protocol family1461worker2 # [ 0.568884] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1462lb # [ 0.535721] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1463lb # [ 0.537211] clocksource: Switched to clocksource acpi_pm1464lb # [ 0.538227] NET: Registered PF_INET protocol family1465lb # [ 0.539424] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1466worker1 # [ 0.558165] pci 0000:00:01.0: vgaarb: setting as boot VGA device1467worker1 # [ 0.558579] pci 0000:00:01.0: vgaarb: bridge control possible1468worker1 # [ 0.558579] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none1469worker1 # [ 0.558587] vgaarb: loaded1470client # [ 0.563339] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1471worker1 # [ 0.559344] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 01472worker1 # [ 0.559585] hpet0: 3 comparators, 64-bit 100.000000 MHz counter1473client # [ 0.564890] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1474client # [ 0.566265] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1475client # [ 0.567666] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1476client # [ 0.568957] TCP: Hash tables configured (established 16384 bind 16384)1477worker2 # [ 0.586199] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1478worker1 # [ 0.564677] clocksource: Switched to clocksource kvm-clock1479client # [ 0.570135] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1480worker2 # [ 0.587827] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1481client # [ 0.571371] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1482worker2 # [ 0.589182] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1483client # [ 0.572488] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1484lb # [ 0.555543] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1485client # [ 0.573698] NET: Registered PF_UNIX/PF_LOCAL protocol family1486worker2 # [ 0.590519] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1487worker1 # [ 0.568719] VFS: Disk quotas dquot_6.6.01488client # [ 0.574656] NET: Registered PF_XDP protocol family1489lb # [ 0.557060] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1490worker2 # [ 0.591784] TCP: Hash tables configured (established 16384 bind 16384)1491worker1 # [ 0.569481] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1492client # [ 0.575479] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1493lb # [ 0.558319] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1494worker1 # [ 0.570898] pnp: PnP ACPI init1495worker2 # [ 0.592916] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1496client # [ 0.576470] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1497worker1 # [ 0.571742] ACPI: IRQ 4 override to edge(!), high(!)1498lb # [ 0.559608] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1499worker2 # [ 0.594185] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1500client # [ 0.577469] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1501lb # [ 0.560844] TCP: Hash tables configured (established 16384 bind 16384)1502worker2 # [ 0.595256] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1503worker1 # [ 0.572746] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved1504client # [ 0.578543] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1505lb # [ 0.561913] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1506worker2 # [ 0.596442] NET: Registered PF_UNIX/PF_LOCAL protocol family1507worker1 # [ 0.574311] pnp: PnP ACPI: found 6 devices1508client # [ 0.579598] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1509worker2 # [ 0.597424] NET: Registered PF_XDP protocol family1510lb # [ 0.563142] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1511client # [ 0.580654] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1512worker2 # [ 0.598280] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1513lb # [ 0.564218] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1514client # [ 0.582380] ACPI: \_SB_.GSIA: Enabled at IRQ 161515worker2 # [ 0.599286] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1516lb # [ 0.565440] NET: Registered PF_UNIX/PF_LOCAL protocol family1517worker2 # [ 0.600283] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1518lb # [ 0.566415] NET: Registered PF_XDP protocol family1519worker2 # [ 0.601397] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1520lb # [ 0.567271] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1521client # [ 0.584642] ACPI: \_SB_.GSIB: Enabled at IRQ 171522lb # [ 0.568267] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1523worker2 # [ 0.602465] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1524client # [ 0.586645] ACPI: \_SB_.GSIC: Enabled at IRQ 181525lb # [ 0.569293] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1526worker2 # [ 0.603607] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1527lb # [ 0.570362] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1528worker2 # [ 0.605279] ACPI: \_SB_.GSIA: Enabled at IRQ 161529worker1 # [ 0.582082] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns1530client # [ 0.588638] ACPI: \_SB_.GSID: Enabled at IRQ 191531lb # [ 0.571455] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1532worker1 # [ 0.583590] clocksource: Switched to clocksource acpi_pm1533worker1 # [ 0.584575] NET: Registered PF_INET protocol family1534lb # [ 0.572500] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1535client # [ 0.590318] PCI: CLS 0 bytes, default 641536worker2 # [ 0.607494] ACPI: \_SB_.GSIB: Enabled at IRQ 171537lb # [ 0.574148] ACPI: \_SB_.GSIA: Enabled at IRQ 161538worker1 # [ 0.585800] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1539client # [ 0.591181] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1540worker2 # [ 0.609809] ACPI: \_SB_.GSIC: Enabled at IRQ 181541client # [ 0.592894] Trying to unpack rootfs image as initramfs...1542lb # [ 0.576294] ACPI: \_SB_.GSIB: Enabled at IRQ 171543worker2 # [ 0.611761] ACPI: \_SB_.GSID: Enabled at IRQ 191544lb # [ 0.578216] ACPI: \_SB_.GSIC: Enabled at IRQ 181545worker2 # [ 0.613416] PCI: CLS 0 bytes, default 641546lb # [ 0.580195] ACPI: \_SB_.GSID: Enabled at IRQ 191547worker2 # [ 0.614298] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1548lb # [ 0.581868] PCI: CLS 0 bytes, default 641549worker2 # [ 0.616025] Trying to unpack rootfs image as initramfs...1550lb # [ 0.582815] Trying to unpack rootfs image as initramfs...1551lb # [ 0.585195] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1552worker1 # [ 0.602807] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1553worker1 # [ 0.604456] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1554worker1 # [ 0.605885] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1555worker1 # [ 0.607370] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1556worker1 # [ 0.608698] TCP: Hash tables configured (established 16384 bind 16384)1557worker1 # [ 0.609917] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1558worker1 # [ 0.611253] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1559worker1 # [ 0.612436] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1560worker1 # [ 0.613754] NET: Registered PF_UNIX/PF_LOCAL protocol family1561worker1 # [ 0.614788] NET: Registered PF_XDP protocol family1562worker1 # [ 0.615693] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]1563worker1 # [ 0.616781] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]1564worker1 # [ 0.617890] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]1565worker1 # [ 0.619077] pci_bus 0000:00: resource 7 [mem 0x60000000-0xafffffff window]1566worker1 # [ 0.620241] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]1567worker1 # [ 0.621427] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window]1568worker1 # [ 0.623190] ACPI: \_SB_.GSIA: Enabled at IRQ 161569worker1 # [ 0.625957] ACPI: \_SB_.GSIB: Enabled at IRQ 171570worker1 # [ 0.627961] ACPI: \_SB_.GSIC: Enabled at IRQ 181571worker1 # [ 0.629981] ACPI: \_SB_.GSID: Enabled at IRQ 191572client # [ 0.636459] Initialise system trusted keyrings1573worker1 # [ 0.631807] PCI: CLS 0 bytes, default 641574client # [ 0.637451] workingset: timestamp_bits=40 max_order=19 bucket_order=01575worker1 # [ 0.632729] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229842dd0bd, max_idle_ns: 440795319571 ns1576worker1 # [ 0.634550] Trying to unpack rootfs image as initramfs...1577lb # [ 0.627156] Initialise system trusted keyrings1578worker2 # [ 0.662108] Initialise system trusted keyrings1579lb # [ 0.630743] workingset: timestamp_bits=40 max_order=19 bucket_order=01580worker2 # [ 0.665733] workingset: timestamp_bits=40 max_order=19 bucket_order=01581client # [ 0.661241] Key type asymmetric registered1582client # [ 0.661960] Asymmetric key parser 'x509' registered1583client # [ 0.662858] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1584client # [ 0.666227] io scheduler mq-deadline registered1585client # [ 0.667009] io scheduler kyber registered1586lb # [ 0.650194] Key type asymmetric registered1587worker2 # [ 0.684963] Key type asymmetric registered1588worker2 # [ 0.687716] Asymmetric key parser 'x509' registered1589lb # [ 0.653709] Asymmetric key parser 'x509' registered1590client # [ 0.671255] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1591worker2 # [ 0.688587] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1592lb # [ 0.654538] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1593client # [ 0.672559] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1594worker2 # [ 0.691841] io scheduler mq-deadline registered1595lb # [ 0.657866] io scheduler mq-deadline registered1596worker2 # [ 0.692601] io scheduler kyber registered1597lb # [ 0.658653] io scheduler kyber registered1598client # [ 0.677861] Linux agpgart interface v0.1031599client # [ 0.678652] ACPI: bus type drm_connector registered1600worker2 # [ 0.696258] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1601lb # [ 0.662299] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1602worker2 # [ 0.697560] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1603lb # [ 0.663569] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1604client # [ 0.683527] usbcore: registered new interface driver usbserial_generic1605worker2 # [ 0.701479] Linux agpgart interface v0.1031606client # [ 0.684657] usbserial: USB Serial support registered for generic1607worker2 # [ 0.702259] ACPI: bus type drm_connector registered1608worker1 # [ 0.679815] Initialise system trusted keyrings1609lb # [ 0.668974] Linux agpgart interface v0.1031610lb # [ 0.669741] ACPI: bus type drm_connector registered1611client # [ 0.688060] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1612worker1 # [ 0.683421] workingset: timestamp_bits=40 max_order=19 bucket_order=01613client # [ 0.688060] Please enable it if your BIOS has the CPPC option.1614worker2 # [ 0.707148] usbcore: registered new interface driver usbserial_generic1615lb # [ 0.673043] usbcore: registered new interface driver usbserial_generic1616client # [ 0.690314] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1617worker2 # [ 0.708215] usbserial: USB Serial support registered for generic1618lb # [ 0.674102] usbserial: USB Serial support registered for generic1619client # [ 0.694186] drop_monitor: Initializing network drop monitor service1620lb # [ 0.676716] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1621worker2 # [ 0.711690] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1622client # [ 0.695370] NET: Registered PF_INET6 protocol family1623lb # [ 0.676716] Please enable it if your BIOS has the CPPC option.1624worker2 # [ 0.711690] Please enable it if your BIOS has the CPPC option.1625lb # [ 0.678936] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1626worker2 # [ 0.713956] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1627client # [ 0.698160] Segment Routing with IPv61628client # [ 0.698816] In-situ OAM (IOAM) with IPv61629worker2 # [ 0.716826] drop_monitor: Initializing network drop monitor service1630lb # [ 0.682829] drop_monitor: Initializing network drop monitor service1631worker2 # [ 0.718017] NET: Registered PF_INET6 protocol family1632lb # [ 0.684008] NET: Registered PF_INET6 protocol family1633client # [ 0.702333] IPI shorthand broadcast: enabled1634worker2 # [ 0.721151] Segment Routing with IPv61635lb # [ 0.687193] Segment Routing with IPv61636worker2 # [ 0.723700] In-situ OAM (IOAM) with IPv61637lb # [ 0.689719] In-situ OAM (IOAM) with IPv61638worker2 # [ 0.724749] IPI shorthand broadcast: enabled1639lb # [ 0.690736] IPI shorthand broadcast: enabled1640worker1 # [ 0.704817] Key type asymmetric registered1641client # [ 0.710626] sched_clock: Marking stable (577014187, 133029035)->(785808231, -75765009)1642worker1 # [ 0.707406] Asymmetric key parser 'x509' registered1643client # [ 0.714228] registered taskstats version 11644worker1 # [ 0.708287] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)1645client # [ 0.715198] Loading compiled-in X.509 certificates1646worker2 # [ 0.732918] sched_clock: Marking stable (595014390, 137654417)->(804390388, -71721581)1647lb # [ 0.698978] sched_clock: Marking stable (566014113, 132673587)->(776046109, -77358409)1648worker1 # [ 0.711497] io scheduler mq-deadline registered1649worker1 # [ 0.712252] io scheduler kyber registered1650lb # [ 0.702923] registered taskstats version 11651worker2 # [ 0.737854] registered taskstats version 11652lb # [ 0.703904] Loading compiled-in X.509 certificates1653worker1 # [ 0.715960] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1654worker2 # [ 0.738944] Loading compiled-in X.509 certificates1655worker1 # [ 0.719383] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A1656worker1 # [ 0.723010] Linux agpgart interface v0.1031657worker1 # [ 0.723852] ACPI: bus type drm_connector registered1658worker1 # [ 0.726924] usbcore: registered new interface driver usbserial_generic1659worker1 # [ 0.728067] usbserial: USB Serial support registered for generic1660client # [ 0.734066] Demotion targets for Node 0: null1661client # [ 0.734924] Key type .fscrypt registered1662client # [ 0.735636] Key type fscrypt-provisioning registered1663client # [ 0.736644] ima: No TPM chip found, activating TPM-bypass!1664worker1 # [ 0.731381] amd_pstate: The CPPC feature is supported but currently disabled by the BIOS.1665worker1 # [ 0.731381] Please enable it if your BIOS has the CPPC option.1666lb # [ 0.721712] Demotion targets for Node 0: null1667worker1 # [ 0.733733] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled1668worker2 # [ 0.756689] Demotion targets for Node 0: null1669client # [ 0.740067] ima: Allocated hash algorithm: sha11670client # [ 0.740835] ima: No architecture policies found1671lb # [ 0.723733] Key type .fscrypt registered1672worker2 # [ 0.758720] Key type .fscrypt registered1673lb # [ 0.724398] Key type fscrypt-provisioning registered1674worker2 # [ 0.759383] Key type fscrypt-provisioning registered1675lb # [ 0.725360] ima: No TPM chip found, activating TPM-bypass!1676client # [ 0.743056] PM: Magic number: 10:312:9871677worker1 # [ 0.737494] drop_monitor: Initializing network drop monitor service1678worker2 # [ 0.760335] ima: No TPM chip found, activating TPM-bypass!1679client # [ 0.743724] machinecheck machinecheck0: hash matches1680worker1 # [ 0.738726] NET: Registered PF_INET6 protocol family1681lb # [ 0.729705] ima: Allocated hash algorithm: sha11682worker1 # [ 0.741890] Segment Routing with IPv61683client # [ 0.747509] RAS: Correctable Errors collector initialized.1684worker2 # [ 0.764684] ima: Allocated hash algorithm: sha11685lb # [ 0.730471] ima: No architecture policies found1686worker2 # [ 0.765457] ima: No architecture policies found1687lb # [ 0.732702] PM: Magic number: 10:312:9871688worker2 # [ 0.767687] PM: Magic number: 10:312:9871689lb # [ 0.733421] machinecheck machinecheck0: hash matches1690worker1 # [ 0.745400] In-situ OAM (IOAM) with IPv61691worker2 # [ 0.768391] machinecheck machinecheck0: hash matches1692worker1 # [ 0.746485] IPI shorthand broadcast: enabled1693lb # [ 0.737102] RAS: Correctable Errors collector initialized.1694worker2 # [ 0.772130] RAS: Correctable Errors collector initialized.1695client # [ 0.756127] clk: Disabling unused clocks1696client # [ 0.756830] PM: genpd: Disabling unused power domains1697worker1 # [ 0.754810] sched_clock: Marking stable (613014872, 141350828)->(835100985, -80735285)1698lb # [ 0.745782] clk: Disabling unused clocks1699worker2 # [ 0.780809] clk: Disabling unused clocks1700lb # [ 0.746487] PM: genpd: Disabling unused power domains1701worker2 # [ 0.781529] PM: genpd: Disabling unused power domains1702worker1 # [ 0.759609] registered taskstats version 11703worker1 # [ 0.760609] Loading compiled-in X.509 certificates1704worker1 # [ 0.778380] Demotion targets for Node 0: null1705worker1 # [ 0.781420] Key type .fscrypt registered1706worker1 # [ 0.782137] Key type fscrypt-provisioning registered1707worker1 # [ 0.783158] ima: No TPM chip found, activating TPM-bypass!1708worker1 # [ 0.787388] ima: Allocated hash algorithm: sha11709worker1 # [ 0.788260] ima: No architecture policies found1710worker1 # [ 0.790384] PM: Magic number: 10:312:9871711worker1 # [ 0.791137] machinecheck machinecheck0: hash matches1712worker1 # [ 0.795015] RAS: Correctable Errors collector initialized.1713worker1 # [ 0.803809] clk: Disabling unused clocks1714worker1 # [ 0.806379] PM: genpd: Disabling unused power domains1715client # [ 0.918474] Freeing initrd memory: 29096K1716client # [ 0.921615] Freeing unused decrypted memory: 2028K1717client # [ 0.924325] Freeing unused kernel image (initmem) memory: 3644K1718client # [ 0.925460] Write protecting the kernel read-only data: 32768k1719lb # [ 0.910454] Freeing initrd memory: 29092K1720client # [ 0.927398] Freeing unused kernel image (text/rodata gap) memory: 1208K1721client # [ 0.928903] Freeing unused kernel image (rodata/data gap) memory: 752K1722lb # [ 0.913591] Freeing unused decrypted memory: 2028K1723lb # [ 0.916166] Freeing unused kernel image (initmem) memory: 3644K1724worker2 # [ 0.951722] Freeing initrd memory: 29092K1725lb # [ 0.917273] Write protecting the kernel read-only data: 32768k1726lb # [ 0.919151] Freeing unused kernel image (text/rodata gap) memory: 1208K1727worker2 # [ 0.954785] Freeing unused decrypted memory: 2028K1728lb # [ 0.920690] Freeing unused kernel image (rodata/data gap) memory: 752K1729worker2 # [ 0.957346] Freeing unused kernel image (initmem) memory: 3644K1730worker2 # [ 0.958437] Write protecting the kernel read-only data: 32768k1731worker2 # [ 0.960351] Freeing unused kernel image (text/rodata gap) memory: 1208K1732worker2 # [ 0.961883] Freeing unused kernel image (rodata/data gap) memory: 752K1733client # [ 0.979936] x86/mm: Checked W+X mappings: passed, no W+X pages found.1734client # [ 0.981100] Run /init as init process1735worker1 # [ 0.978230] Freeing initrd memory: 29092K1736worker1 # [ 0.981502] Freeing unused decrypted memory: 2028K1737lb # [ 0.971703] x86/mm: Checked W+X mappings: passed, no W+X pages found.1738worker1 # [ 0.984254] Freeing unused kernel image (initmem) memory: 3644K1739lb # [ 0.972827] Run /init as init process1740worker1 # [ 0.985333] Write protecting the kernel read-only data: 32768k1741client # [ 0.991239] systemd[1]: Inserted module 'autofs4'1742worker1 # [ 0.987245] Freeing unused kernel image (text/rodata gap) memory: 1208K1743worker1 # [ 0.988867] Freeing unused kernel image (rodata/data gap) memory: 752K1744worker2 # [ 1.013245] x86/mm: Checked W+X mappings: passed, no W+X pages found.1745worker2 # [ 1.014371] Run /init as init process1746lb # [ 0.983086] systemd[1]: Inserted module 'autofs4'1747client # [ 1.007408] fuse: init (API version 7.45)1748worker2 # [ 1.024294] systemd[1]: Inserted module 'autofs4'1749client # [ 1.013193] ACPI: \_SB_.GSIG: Enabled at IRQ 221750client # [ 1.015490] ACPI: \_SB_.GSIH: Enabled at IRQ 231751lb # [ 0.999313] fuse: init (API version 7.45)1752client # [ 1.018629] ACPI: \_SB_.GSIE: Enabled at IRQ 201753client # [ 1.020876] ACPI: \_SB_.GSIF: Enabled at IRQ 211754lb # [ 1.005023] ACPI: \_SB_.GSIG: Enabled at IRQ 221755worker2 # [ 1.040401] fuse: init (API version 7.45)1756lb # [ 1.007411] ACPI: \_SB_.GSIH: Enabled at IRQ 231757lb # [ 1.011171] ACPI: \_SB_.GSIE: Enabled at IRQ 201758worker2 # [ 1.046090] ACPI: \_SB_.GSIG: Enabled at IRQ 221759lb # [ 1.013946] ACPI: \_SB_.GSIF: Enabled at IRQ 211760worker2 # [ 1.048486] ACPI: \_SB_.GSIH: Enabled at IRQ 231761worker2 # [ 1.051483] ACPI: \_SB_.GSIE: Enabled at IRQ 201762worker2 # [ 1.053690] ACPI: \_SB_.GSIF: Enabled at IRQ 211763worker1 # [ 1.040311] x86/mm: Checked W+X mappings: passed, no W+X pages found.1764worker1 # [ 1.041506] Run /init as init process1765client # [ 1.046851] systemd[1]: Successfully made /usr/ read-only.1766worker1 # [ 1.052487] systemd[1]: Inserted module 'autofs4'1767lb # [ 1.040630] systemd[1]: Successfully made /usr/ read-only.1768worker2 # [ 1.083823] systemd[1]: Successfully made /usr/ read-only.1769worker1 # [ 1.068808] fuse: init (API version 7.45)1770worker1 # [ 1.074740] ACPI: \_SB_.GSIG: Enabled at IRQ 221771worker1 # [ 1.077583] ACPI: \_SB_.GSIH: Enabled at IRQ 231772worker1 # [ 1.080976] ACPI: \_SB_.GSIE: Enabled at IRQ 201773worker1 # [ 1.083492] ACPI: \_SB_.GSIF: Enabled at IRQ 211774worker1 # [ 1.111224] systemd[1]: Successfully made /usr/ read-only.1775client # [ 1.383350] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1776client # [ 1.395230] systemd[1]: Detected virtualization kvm.1777client # [ 1.397310] systemd[1]: Detected architecture x86-64.1778client # [ 1.399423] systemd[1]: Running in initrd.1779client # [ 1.401824] systemd[1]: Initializing machine ID from random generator.1780lb # [ 1.377033] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1781client # [ 1.404667] systemd[1]: Hostname set to <client>.1782lb # [ 1.389158] systemd[1]: Detected virtualization kvm.1783lb # [ 1.391366] systemd[1]: Detected architecture x86-64.1784lb # [ 1.393579] systemd[1]: Running in initrd.1785worker2 # [ 1.420796] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1786worker2 # [ 1.430781] systemd[1]: Detected virtualization kvm.1787lb # [ 1.396164] systemd[1]: Initializing machine ID from random generator.1788worker2 # [ 1.432546] systemd[1]: Detected architecture x86-64.1789lb # [ 1.399109] systemd[1]: Hostname set to <lb>.1790worker2 # [ 1.434314] systemd[1]: Running in initrd.1791worker2 # [ 1.436403] systemd[1]: Initializing machine ID from random generator.1792worker2 # [ 1.438810] systemd[1]: Hostname set to <worker2>.1793worker1 # [ 1.446780] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1794worker1 # [ 1.451904] systemd[1]: Detected virtualization kvm.1795worker1 # [ 1.452796] systemd[1]: Detected architecture x86-64.1796worker1 # [ 1.453752] systemd[1]: Running in initrd.1797worker1 # [ 1.454886] systemd[1]: Initializing machine ID from random generator.1798worker1 # [ 1.456175] systemd[1]: Hostname set to <worker1>.1799client # [ 1.623486] systemd[1]: bpf-restrict-fs: LSM BPF program attached1800lb # [ 1.623196] systemd[1]: bpf-restrict-fs: LSM BPF program attached1801worker2 # [ 1.661095] systemd[1]: bpf-restrict-fs: LSM BPF program attached1802worker1 # [ 1.665569] systemd[1]: bpf-restrict-fs: LSM BPF program attached1803client # [ 1.678865] systemd[1]: Queued start job for default target Initrd Default Target.1804client # [ 1.683327] systemd[1]: Created slice Slice /system/modprobe.1805client # [ 1.684524] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1806client # [ 1.685982] systemd[1]: Expecting device /dev/disk/by-label/nixos...1807client # [ 1.687222] systemd[1]: Reached target Path Units.1808client # [ 1.688116] systemd[1]: Reached target Slice Units.1809client # [ 1.688973] systemd[1]: Reached target Swaps.1810client # [ 1.689787] systemd[1]: Reached target Timer Units.1811client # [ 1.690751] systemd[1]: Listening on D-Bus System Message Bus Socket.1812client # [ 1.691969] systemd[1]: Listening on Journal Socket (/dev/log).1813client # [ 1.693177] systemd[1]: Listening on Journal Sockets.1814client # [ 1.694180] systemd[1]: Listening on udev Control Socket.1815lb # [ 1.677328] systemd[1]: Queued start job for default target Initrd Default Target.1816client # [ 1.695234] systemd[1]: Listening on udev Kernel Socket.1817client # [ 1.696187] systemd[1]: Reached target Socket Units.1818client # [ 1.697957] systemd[1]: Starting Create List of Static Device Nodes...1819lb # [ 1.681027] systemd[1]: Created slice Slice /system/modprobe.1820lb # [ 1.682210] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1821worker2 # [ 1.716584] systemd[1]: Queued start job for default target Initrd Default Target.1822lb # [ 1.683616] systemd[1]: Expecting device /dev/disk/by-label/nixos...1823lb # [ 1.684776] systemd[1]: Reached target Path Units.1824client # [ 1.701846] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1825lb # [ 1.685592] systemd[1]: Reached target Slice Units.1826lb # [ 1.686476] systemd[1]: Reached target Swaps.1827worker2 # [ 1.721077] systemd[1]: Created slice Slice /system/modprobe.1828lb # [ 1.687287] systemd[1]: Reached target Timer Units.1829worker2 # [ 1.722230] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1830lb # [ 1.688272] systemd[1]: Listening on D-Bus System Message Bus Socket.1831worker2 # [ 1.723587] systemd[1]: Expecting device /dev/disk/by-label/nixos...1832lb # [ 1.689481] systemd[1]: Listening on Journal Socket (/dev/log).1833worker2 # [ 1.724738] systemd[1]: Reached target Path Units.1834lb # [ 1.690637] systemd[1]: Listening on Journal Sockets.1835worker2 # [ 1.725560] systemd[1]: Reached target Slice Units.1836lb # [ 1.691626] systemd[1]: Listening on udev Control Socket.1837worker2 # [ 1.726395] systemd[1]: Reached target Swaps.1838client # [ 1.709272] systemd[1]: Starting Load Kernel Module configfs...1839lb # [ 1.692645] systemd[1]: Listening on udev Kernel Socket.1840worker2 # [ 1.727192] systemd[1]: Reached target Timer Units.1841lb # [ 1.693580] systemd[1]: Reached target Socket Units.1842worker2 # [ 1.728146] systemd[1]: Listening on D-Bus System Message Bus Socket.1843worker2 # [ 1.729319] systemd[1]: Listening on Journal Socket (/dev/log).1844lb # [ 1.695312] systemd[1]: Starting Create List of Static Device Nodes...1845worker2 # [ 1.730421] systemd[1]: Listening on Journal Sockets.1846worker2 # [ 1.731386] systemd[1]: Listening on udev Control Socket.1847worker2 # [ 1.732360] systemd[1]: Listening on udev Kernel Socket.1848worker2 # [ 1.733263] systemd[1]: Reached target Socket Units.1849lb # [ 1.699759] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1850worker2 # [ 1.735129] systemd[1]: Starting Create List of Static Device Nodes...1851worker2 # [ 1.739741] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1852client # [ 1.723118] systemd[1]: Starting Journal Service...1853lb # [ 1.706764] systemd[1]: Starting Load Kernel Module configfs...1854worker1 # [ 1.719152] systemd[1]: Queued start job for default target Initrd Default Target.1855worker1 # [ 1.723711] systemd[1]: Created slice Slice /system/modprobe.1856worker2 # [ 1.746864] systemd[1]: Starting Load Kernel Module configfs...1857worker1 # [ 1.724998] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1858worker1 # [ 1.726509] systemd[1]: Expecting device /dev/disk/by-label/nixos...1859worker1 # [ 1.727678] systemd[1]: Reached target Path Units.1860worker1 # [ 1.728587] systemd[1]: Reached target Slice Units.1861worker1 # [ 1.729509] systemd[1]: Reached target Swaps.1862worker1 # [ 1.730312] systemd[1]: Reached target Timer Units.1863lb # [ 1.719660] systemd[1]: Starting Journal Service...1864worker1 # [ 1.731300] systemd[1]: Listening on D-Bus System Message Bus Socket.1865worker1 # [ 1.732573] systemd[1]: Listening on Journal Socket (/dev/log).1866worker1 # [ 1.733805] systemd[1]: Listening on Journal Sockets.1867worker1 # [ 1.734861] systemd[1]: Listening on udev Control Socket.1868worker1 # [ 1.736019] systemd[1]: Listening on udev Kernel Socket.1869worker1 # [ 1.737032] systemd[1]: Reached target Socket Units.1870worker2 # [ 1.760859] systemd[1]: Starting Journal Service...1871worker1 # [ 1.738963] systemd[1]: Starting Create List of Static Device Nodes...1872client # [ 1.733736] systemd[1]: Starting Load Kernel Modules...1873worker1 # [ 1.743460] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1874client # [ 1.754534] netfs: FS-Cache loaded1875worker1 # [ 1.750443] systemd[1]: Starting Load Kernel Module configfs...1876client # [ 1.759210] 9pnet: Installing 9P2000 support1877lb # [ 1.730781] systemd[1]: Starting Load Kernel Modules...1878lb # [ 1.751728] netfs: FS-Cache loaded1879worker2 # [ 1.774738] systemd[1]: Starting Load Kernel Modules...1880client # [ 1.769114] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1881worker1 # [ 1.765555] systemd[1]: Starting Journal Service...1882lb # [ 1.757300] 9pnet: Installing 9P2000 support1883client # [ 1.776145] systemd[1]: Starting Coldplug All udev Devices...1884worker2 # [ 1.794415] netfs: FS-Cache loaded1885worker2 # [ 1.800299] 9pnet: Installing 9P2000 support1886lb # [ 1.765879] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1887client # [ 1.788238] systemd[1]: Finished Create List of Static Device Nodes.1888worker2 # [ 1.805866] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1889lb # [ 1.773950] systemd[1]: Starting Coldplug All udev Devices...1890client # [ 1.795655] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1891worker1 # [ 1.779499] systemd[1]: Starting Load Kernel Modules...1892worker2 # [ 1.815763] systemd[1]: Starting Coldplug All udev Devices...1893client # [ 1.798756] systemd-journald[67]: Collecting audit messages is disabled.1894worker1 # [ 1.798781] netfs: FS-Cache loaded1895lb # [ 1.787171] systemd[1]: Finished Create List of Static Device Nodes.1896client # [ 1.805192] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1897worker1 # [ 1.804169] 9pnet: Installing 9P2000 support1898lb # [ 1.792292] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1899client # [ 1.810461] systemd[1]: modprobe@configfs.service: Deactivated successfully.1900lb # [ 1.796380] systemd-journald[67]: Collecting audit messages is disabled.1901worker2 # [ 1.830821] systemd[1]: Finished Create List of Static Device Nodes.1902client # [ 1.814778] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1903lb # [ 1.801060] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1904client # [ 1.818411] systemd[1]: Finished Load Kernel Module configfs.1905worker1 # [ 1.812892] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1906worker2 # [ 1.838338] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1907lb # [ 1.805910] systemd[1]: modprobe@configfs.service: Deactivated successfully.1908client # [ 1.822511] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1909worker1 # [ 1.821467] systemd[1]: Starting Coldplug All udev Devices...1910worker2 # [ 1.844410] systemd-journald[67]: Collecting audit messages is disabled.1911client # [ 1.827184] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1912lb # [ 1.811803] systemd[1]: Finished Load Kernel Module configfs.1913worker2 # [ 1.847058] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1914lb # [ 1.813615] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1915worker2 # [ 1.851128] systemd[1]: modprobe@configfs.service: Deactivated successfully.1916client # [ 1.835129] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1917lb # [ 1.818153] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1918worker2 # [ 1.859022] systemd[1]: Finished Load Kernel Module configfs.1919lb # [ 1.824863] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1920worker1 # [ 1.837690] systemd[1]: Finished Create List of Static Device Nodes.1921worker2 # [ 1.863149] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1922lb # [ 1.829757] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1923worker1 # [ 1.842139] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1924worker2 # [ 1.865707] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1925worker1 # [ 1.849254] systemd-journald[67]: Collecting audit messages is disabled.1926client # [ 1.856172] systemd[1]: Finished Load Kernel Modules.1927worker1 # [ 1.850936] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1928worker2 # [ 1.873834] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1929worker2 # [ 1.876773] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1930worker1 # [ 1.856793] systemd[1]: modprobe@configfs.service: Deactivated successfully.1931client # [ 1.863283] systemd[1]: Starting Apply Kernel Variables...1932worker1 # [ 1.865405] systemd[1]: Finished Load Kernel Module configfs.1933lb # [ 1.854757] systemd[1]: Finished Load Kernel Modules.1934worker1 # [ 1.867638] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1935client # [ 1.876226] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1936worker1 # [ 1.871921] systemd[1]: Kernel Configuration File System skipped, unmet condition check ConditionPathExists=/sys/kernel/config1937lb # [ 1.862790] systemd[1]: Starting Apply Kernel Variables...1938worker1 # [ 1.877553] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1939client # [ 1.885704] systemd[1]: Starting Create Static Device Nodes in /dev...1940worker2 # [ 1.904242] systemd[1]: Finished Load Kernel Modules.1941worker1 # [ 1.882448] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1942worker2 # [ 1.910855] systemd[1]: Starting Apply Kernel Variables...1943lb # [ 1.877768] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1944lb # [ 1.885897] systemd[1]: Starting Create Static Device Nodes in /dev...1945worker2 # [ 1.921818] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1946client # [ 1.905365] systemd[1]: Finished Apply Kernel Variables.1947worker1 # [ 1.907470] systemd[1]: Finished Load Kernel Modules.1948worker2 # [ 1.931352] systemd[1]: Starting Create Static Device Nodes in /dev...1949lb # [ 1.898767] systemd[1]: Finished Apply Kernel Variables.1950client # [ 1.783674] systemd-modules-load[68]: Inserted module 'dm_mod'[ 1.917759] systemd[1]: Started Journal Service.1951client # 1952worker1 # [ 1.915585] systemd[1]: Starting Apply Kernel Variables...1953client # [ 1.788324] systemd-modules-load[68]: Inserted module 'virtio_balloon'1954client # [ 1.789419] systemd-modules-load[68]: Inserted module 'virtio_gpu'1955client # [ 1.790501] systemd[1]: Finished Create Static Device Nodes in /dev.1956client # [ 1.792348] systemd[1]: Reached target Preparation for Local File Systems.1957client # [ 1.794334] systemd[1]: Reached target Local File Systems.1958client # [ 1.796281] systemd[1]: Starting Create System Files and Directories...1959lb # [ 1.782479] systemd-modules-load[68]: Inserted module 'dm_mod'[ 1.916259] systemd[1]: Started Journal Service.1960lb # 1961client # [ 1.801199] systemd[1]: Starting Rule-based Manager for Device Events and Files...1962lb # [ 1.785300] systemd-modules-load[68]: Inserted module 'virtio_balloon'1963worker2 # [ 1.952234] systemd[1]: Finished Apply Kernel Variables.1964worker1 # [ 1.930522] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1965lb # [ 1.788180] systemd-modules-load[68]: Inserted module 'virtio_gpu'1966lb # [ 1.795119] systemd[1]: Finished Create Static Device Nodes in /dev.1967worker1 # [ 1.940574] systemd[1]: Starting Create Static Device Nodes in /dev...1968worker2 # [ 1.964266] systemd[1]: Started Journal Service.1969lb # [ 1.799680] systemd[1]: Reached target Preparation for Local File Systems.1970lb # [ 1.801103] systemd[1]: Reached target Local File Systems.1971worker2 # [ 1.828222] systemd-modules-load[68]: Inserted module 'dm_mod'1972worker2 # [ 1.832442] systemd-modules-load[68]: Inserted module 'virtio_balloon'1973lb # [ 1.803277] systemd[1]: Starting Create System Files and Directories...1974worker2 # [ 1.833574] systemd-modules-load[68]: Inserted module 'virtio_gpu'1975worker2 # [ 1.836076] systemd[1]: Finished Create Static Device Nodes in /dev.1976client # [ 1.824444] systemd[1]: Finished Create System Files and Directories.1977worker2 # [ 1.837252] systemd[1]: Reached target Preparation for Local File Systems.1978worker2 # [ 1.839315] systemd[1]: Reached target Local File Systems.1979worker1 # [ 1.954492] systemd[1]: Finished Apply Kernel Variables.1980lb # [ 1.810384] systemd[1]: Starting Rule-based Manager for Device Events and Files...1981worker2 # [ 1.841257] systemd[1]: Starting Create System Files and Directories...1982worker2 # [ 1.847054] systemd[1]: Starting Rule-based Manager for Device Events and Files...1983worker1 # [ 1.972979] systemd[1]: Started Journal Service.1984worker1 # [ 1.832691] systemd-modules-load[68]: Inserted module 'dm_mod'1985lb # [ 1.832377] systemd[1]: Finished Create System Files and Directories.1986worker1 # [ 1.835905] systemd-modules-load[68]: Inserted module 'virtio_balloon'1987worker1 # [ 1.836940] systemd-modules-load[68]: Inserted module 'virtio_gpu'1988client # [ 1.854732] systemd-udevd[76]: Using default interface naming scheme 'v261'.1989worker1 # [ 1.842681] systemd[1]: Finished Create Static Device Nodes in /dev.1990worker2 # [ 1.872651] systemd[1]: Finished Create System Files and Directories.1991worker1 # [ 1.850116] systemd[1]: Reached target Preparation for Local File Systems.1992worker1 # [ 1.851119] systemd[1]: Reached target Local File Systems.1993worker1 # [ 1.854075] systemd[1]: Starting Create System Files and Directories...1994worker1 # [ 1.859079] systemd[1]: Starting Rule-based Manager for Device Events and Files...1995lb # [ 1.857594] systemd-udevd[84]: Using default interface naming scheme 'v261'.1996client # [ 1.879094] systemd[1]: Started Rule-based Manager for Device Events and Files.1997worker2 # [ 1.898854] systemd-udevd[76]: Using default interface naming scheme 'v261'.1998worker1 # [ 1.886439] systemd[1]: Finished Create System Files and Directories.1999lb # [ 1.884071] systemd[1]: Started Rule-based Manager for Device Events and Files.2000worker2 # [ 1.923298] systemd[1]: Started Rule-based Manager for Device Events and Files.
loading 2,317 hidden lines…
4318worker1 # [ 28.147838] nix-grpc-daemon[784]: ts=2026-09-21T12:57:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4319worker2 # [ 28.231564] nix-grpc-daemon[788]: ts=2026-09-21T12:57:57Z level=info event=scheduler_connected addr=lb:500514320worker2 # [ 28.235254] nix-grpc-daemon[788]: ts=2026-09-21T12:57:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4321worker1 # [ 28.350303] nix-grpc-daemon[784]: ts=2026-09-21T12:57:57Z level=info event=scheduler_connected addr=lb:500514322worker1 # [ 28.353478] nix-grpc-daemon[784]: ts=2026-09-21T12:57:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4323worker2 # [ 28.436474] nix-grpc-daemon[788]: ts=2026-09-21T12:57:57Z level=info event=scheduler_connected addr=lb:500514324worker2 # [ 28.440251] nix-grpc-daemon[788]: ts=2026-09-21T12:57:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4325worker1 # [ 28.551521] nix-grpc-daemon[784]: ts=2026-09-21T12:57:57Z level=info event=scheduler_connected addr=lb:500514326worker1 # [ 28.555469] nix-grpc-daemon[784]: ts=2026-09-21T12:57:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4327worker2 # [ 28.641335] nix-grpc-daemon[788]: ts=2026-09-21T12:57:57Z level=info event=scheduler_connected addr=lb:500514328worker2 # [ 28.645148] nix-grpc-daemon[788]: ts=2026-09-21T12:57:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4329worker1 # [ 28.756457] nix-grpc-daemon[784]: ts=2026-09-21T12:57:57Z level=info event=scheduler_connected addr=lb:500514330worker1 # [ 28.762137] nix-grpc-daemon[784]: ts=2026-09-21T12:57:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4331worker2 # [ 28.845893] nix-grpc-daemon[788]: ts=2026-09-21T12:57:57Z level=info event=scheduler_connected addr=lb:500514332worker2 # [ 28.849635] nix-grpc-daemon[788]: ts=2026-09-21T12:57:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4333worker1 # [ 28.958118] nix-grpc-daemon[784]: ts=2026-09-21T12:57:57Z level=info event=scheduler_connected addr=lb:500514334worker1 # [ 28.962504] nix-grpc-daemon[784]: ts=2026-09-21T12:57:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4335worker2 # [ 29.051864] nix-grpc-daemon[788]: ts=2026-09-21T12:57:58Z level=info event=scheduler_connected addr=lb:500514336worker2 # [ 29.054939] nix-grpc-daemon[788]: ts=2026-09-21T12:57:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4337worker1 # [ 29.163504] nix-grpc-daemon[784]: ts=2026-09-21T12:57:58Z level=info event=scheduler_connected addr=lb:500514338worker1 # [ 29.167943] nix-grpc-daemon[784]: ts=2026-09-21T12:57:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4339worker2 # [ 29.253258] nix-grpc-daemon[788]: ts=2026-09-21T12:57:58Z level=info event=scheduler_connected addr=lb:500514340worker2 # [ 29.256993] nix-grpc-daemon[788]: ts=2026-09-21T12:57:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4341worker1 # [ 29.370341] nix-grpc-daemon[784]: ts=2026-09-21T12:57:58Z level=info event=scheduler_connected addr=lb:500514342worker1 # [ 29.373755] nix-grpc-daemon[784]: ts=2026-09-21T12:57:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4343worker2 # [ 29.457984] nix-grpc-daemon[788]: ts=2026-09-21T12:57:58Z level=info event=scheduler_connected addr=lb:500514344worker2 # [ 29.461822] nix-grpc-daemon[788]: ts=2026-09-21T12:57:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4345worker1 # [ 29.571105] nix-grpc-daemon[784]: ts=2026-09-21T12:57:58Z level=info event=scheduler_connected addr=lb:500514346worker1 # [ 29.576451] nix-grpc-daemon[784]: ts=2026-09-21T12:57:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4347worker2 # [ 29.662833] nix-grpc-daemon[788]: ts=2026-09-21T12:57:58Z level=info event=scheduler_connected addr=lb:500514348worker2 # [ 29.666803] nix-grpc-daemon[788]: ts=2026-09-21T12:57:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4349worker1 # [ 29.777150] nix-grpc-daemon[784]: ts=2026-09-21T12:57:58Z level=info event=scheduler_connected addr=lb:500514350worker1 # [ 29.781489] nix-grpc-daemon[784]: ts=2026-09-21T12:57:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4351worker2 # [ 29.867580] nix-grpc-daemon[788]: ts=2026-09-21T12:57:58Z level=info event=scheduler_connected addr=lb:500514352worker2 # [ 29.871321] nix-grpc-daemon[788]: ts=2026-09-21T12:57:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4353worker1 # [ 29.982533] nix-grpc-daemon[784]: ts=2026-09-21T12:57:58Z level=info event=scheduler_connected addr=lb:500514354worker1 # [ 29.986973] nix-grpc-daemon[784]: ts=2026-09-21T12:57:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4355worker2 # [ 30.073841] nix-grpc-daemon[788]: ts=2026-09-21T12:57:59Z level=info event=scheduler_connected addr=lb:500514356worker2 # [ 30.076908] nix-grpc-daemon[788]: ts=2026-09-21T12:57:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4357worker1 # [ 30.187884] nix-grpc-daemon[784]: ts=2026-09-21T12:57:59Z level=info event=scheduler_connected addr=lb:500514358worker1 # [ 30.192396] nix-grpc-daemon[784]: ts=2026-09-21T12:57:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4359worker2 # [ 30.275301] nix-grpc-daemon[788]: ts=2026-09-21T12:57:59Z level=info event=scheduler_connected addr=lb:500514360worker2 # [ 30.279156] nix-grpc-daemon[788]: ts=2026-09-21T12:57:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4361worker1 # [ 30.394972] nix-grpc-daemon[784]: ts=2026-09-21T12:57:59Z level=info event=scheduler_connected addr=lb:500514362worker1 # [ 30.398669] nix-grpc-daemon[784]: ts=2026-09-21T12:57:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4363worker2 # [ 30.480386] nix-grpc-daemon[788]: ts=2026-09-21T12:57:59Z level=info event=scheduler_connected addr=lb:500514364worker2 # [ 30.484242] nix-grpc-daemon[788]: ts=2026-09-21T12:57:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4365worker1 # [ 30.596146] nix-grpc-daemon[784]: ts=2026-09-21T12:57:59Z level=info event=scheduler_connected addr=lb:500514366worker1 # [ 30.600576] nix-grpc-daemon[784]: ts=2026-09-21T12:57:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4367worker2 # [ 30.685296] nix-grpc-daemon[788]: ts=2026-09-21T12:57:59Z level=info event=scheduler_connected addr=lb:500514368worker2 # [ 30.689141] nix-grpc-daemon[788]: ts=2026-09-21T12:57:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4369worker1 # [ 30.801314] nix-grpc-daemon[784]: ts=2026-09-21T12:57:59Z level=info event=scheduler_connected addr=lb:500514370worker1 # [ 30.805569] nix-grpc-daemon[784]: ts=2026-09-21T12:57:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4371worker2 # [ 30.890059] nix-grpc-daemon[788]: ts=2026-09-21T12:57:59Z level=info event=scheduler_connected addr=lb:500514372worker2 # [ 30.893809] nix-grpc-daemon[788]: ts=2026-09-21T12:57:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4373worker1 # [ 31.006599] nix-grpc-daemon[784]: ts=2026-09-21T12:57:59Z level=info event=scheduler_connected addr=lb:500514374worker1 # [ 31.011073] nix-grpc-daemon[784]: ts=2026-09-21T12:57:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4375lb # [ 31.029976] rustfs-bucket-start[1060]: mb s3://farm4376lb # [ 31.035461] systemd[1]: Finished rustfs-bucket.service.4377lb # [ 31.041906] systemd[1]: Starting niks3 server...4378worker2 # [ 31.096102] nix-grpc-daemon[788]: ts=2026-09-21T12:58:00Z level=info event=scheduler_connected addr=lb:500514379worker2 # [ 31.099329] nix-grpc-daemon[788]: ts=2026-09-21T12:58:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4380worker1 # [ 31.212076] nix-grpc-daemon[784]: ts=2026-09-21T12:58:00Z level=info event=scheduler_connected addr=lb:500514381worker1 # [ 31.216418] nix-grpc-daemon[784]: ts=2026-09-21T12:58:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4382worker2 # [ 31.297126] nix-grpc-daemon[788]: ts=2026-09-21T12:58:00Z level=info event=scheduler_connected addr=lb:500514383worker2 # [ 31.301102] nix-grpc-daemon[788]: ts=2026-09-21T12:58:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4384lb # [ 31.303706] postgres[1075]: [1075] ERROR: relation "goose_db_version" does not exist at character 364385lb # [ 31.304988] postgres[1075]: [1075] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC4386lb # [ 31.345151] niks3-server[1069]: 2026/09/21 12:58:00 OK 20241026095416_initial_model.sql (21.65ms)4387lb # [ 31.348833] niks3-server[1069]: 2026/09/21 12:58:00 OK 20251210153512_drop_unused_gin_index.sql (2.36ms)4388lb # [ 31.353968] niks3-server[1069]: 2026/09/21 12:58:00 OK 20251218171726_add_pins.sql (5.11ms)4389lb # [ 31.359149] niks3-server[1069]: 2026/09/21 12:58:00 OK 20260628120000_add_object_size_and_stats.sql (5.22ms)4390lb # [ 31.364263] niks3-server[1069]: 2026/09/21 12:58:00 OK 20260905000000_add_claims.sql (3.64ms)4391lb # [ 31.369032] niks3-server[1069]: 2026/09/21 12:58:00 OK 20260920000000_drop_claims.sql (4.68ms)4392lb # [ 31.370189] niks3-server[1069]: 2026/09/21 12:58:00 goose: successfully migrated database to version: 202609200000004393lb # [ 31.376110] niks3-server[1069]: 2026/09/21 12:58:00 OK 1_commit_pending_closure.sql (7.05ms)4394lb # [ 31.378703] niks3-server[1069]: 2026/09/21 12:58:00 OK 2_object_stats_trigger.sql (2.63ms)4395lb # [ 31.379918] niks3-server[1069]: 2026/09/21 12:58:00 goose: up to current file version: 24396lb # [ 31.387320] niks3-server[1069]: 2026/09/21 12:58:00 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key4397worker1 # [ 31.418810] nix-grpc-daemon[784]: ts=2026-09-21T12:58:00Z level=info event=scheduler_connected addr=lb:500514398worker1 # [ 31.422181] nix-grpc-daemon[784]: ts=2026-09-21T12:58:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4399worker2 # [ 31.501860] nix-grpc-daemon[788]: ts=2026-09-21T12:58:00Z level=info event=scheduler_connected addr=lb:500514400lb # [ 31.476064] niks3-server[1069]: 2026/09/21 12:58:00 INFO Read proxy enabled — serving cache objects from S34401lb # [ 31.477842] niks3-server[1069]: 2026/09/21 12:58:00 INFO Using socket-activated listener address=0.0.0.0:57514402worker2 # [ 31.505801] nix-grpc-daemon[788]: ts=2026-09-21T12:58:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4403lb # [ 31.482372] niks3-server[1069]: 2026/09/21 12:58:00 INFO systemd watchdog enabled interval=15s4404lb # [ 31.483525] systemd[1]: Started niks3 server.4405lb # [ 31.484200] systemd[1]: Reached target Multi-User System.4406lb # [ 31.485903] systemd[1]: Startup finished in 843ms (kernel) + 4.160s (initrd) + 26.481s (userspace) = 31.485s.4407lb # [ 31.488123] niks3-server[1069]: 2026/09/21 12:58:00 INFO Starting HTTP server address=0.0.0.0:57514408lb # [ 31.494926] niks3-server[1069]: 2026/09/21 12:58:00 INFO lead: acquired remote=192.168.1.4:444804409worker2 # [ 31.530788] nix-grpc-daemon[788]: ts=2026-09-21T12:58:00Z level=info event=scheduler_take_over4410worker1 # [ 31.619759] nix-grpc-daemon[784]: ts=2026-09-21T12:58:00Z level=info event=scheduler_connected addr=lb:500514411worker1 # [ 31.623833] nix-grpc-daemon[784]: ts=2026-09-21T12:58:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4412worker2 # [ 31.706834] nix-grpc-daemon[788]: ts=2026-09-21T12:58:00Z level=info event=scheduler_connected addr=lb:500514413worker2 # [ 31.710788] nix-grpc-daemon[788]: ts=2026-09-21T12:58:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4414worker1 # [ 31.824574] nix-grpc-daemon[784]: ts=2026-09-21T12:58:00Z level=info event=scheduler_connected addr=lb:500514415worker2 # [ 31.865738] nix-grpc-daemon[788]: ts=2026-09-21T12:58:00Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:441364416worker2 # [ 31.868445] nix-grpc-daemon[788]: ts=2026-09-21T12:58:00Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04417worker2 # [ 31.911752] nix-grpc-daemon[788]: ts=2026-09-21T12:58:00Z level=info event=scheduler_connected addr=lb:500514418worker2 # [ 31.937636] nix-grpc-daemon[788]: ts=2026-09-21T12:58:01Z level=info event=oidc_jwks provider=mock keys=14419worker2 # [ 31.942223] nix-grpc-daemon[788]: ts=2026-09-21T12:58:01Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:441364420worker2 # [ 31.946211] nix-grpc-daemon[788]: ts=2026-09-21T12:58:01Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=04421lb: (finished: waiting for unit niks3.service, in 20.93 seconds)4422lb: waiting for TCP port 5751 on localhost4423lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!4424lb: (finished: waiting for TCP port 5751 on localhost, in 0.08 seconds)4425lb: waiting for unit envoy.service4426lb: (finished: waiting for unit envoy.service, in 0.06 seconds)4427lb: waiting for TCP port 50051 on localhost4428lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!4429lb # [ 32.167942] envoy[922]: conn peer=[::1]:38628 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=34430lb: (finished: waiting for TCP port 50051 on localhost, in 0.04 seconds)4431worker2 # [ 32.332183] systemd[1]: Starting scheduler-token.service...4432worker2 # [ 32.400687] systemd[1]: scheduler-token.service: Deactivated successfully.4433worker2 # [ 32.403523] systemd[1]: Finished scheduler-token.service.4434worker2 # [ 32.404929] systemd[1]: scheduler-token.service: Consumed 19ms CPU time over 70ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4435lb: waiting until exactly no endpoints of x86_64-linux are unhealthy4436lb: must succeed: curl -sf localhost:9901/clusters4437lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)4438lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 0.09 seconds)4439subtest: one node schedules and both builders hold a WorkerSession on it4440lb: must succeed: curl -sf localhost:9901/clusters4441lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4442worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4443worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4444lb: must succeed: curl -sf localhost:9901/clusters4445lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4446lb: must succeed: curl -sf localhost:9901/clusters4447lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4448lb: must succeed: curl -sf localhost:9901/clusters4449lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4450lb: must succeed: curl -sf localhost:9901/clusters4451lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4452worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4453worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.03 seconds)4454worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4455worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4456lb: must succeed: curl -sf localhost:9901/clusters4457lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4458lb: must succeed: curl -sf localhost:9901/clusters4459lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4460worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4461worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4462worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4463worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4464worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4465worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4466worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4467worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4468(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.65 seconds)4469subtest: balancer auth: client cert, bearer token, nothing, unknown CN4470worker1 # [ 33.528578] nix-grpc-daemon[784]: ts=2026-09-21T12:58:02Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:544224471worker1 # [ 33.541826] systemd[1]: Started Nix Daemon.4472worker1 # [ 33.788747] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4473worker1 # [ 33.821580] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 16 ms (attempt 1/5)4474worker1 # [ 33.840586] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 186 ms (attempt 2/5)4475worker1 # [ 34.027110] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 142 ms (attempt 3/5)4476worker1 # [ 34.169704] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 184 ms (attempt 4/5)4477worker1 # [ 34.357594] nix-grpc-daemon[784]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4478worker1 # [ 34.362682] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4479worker2 # [ 34.429498] nix-grpc-daemon[788]: ts=2026-09-21T12:58:03Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:546064480worker2 # [ 34.436306] systemd[1]: Started Nix Daemon.4481worker2 # [ 34.630333] nix-daemon[873]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4482worker2 # [ 34.655457] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 79 ms (attempt 1/5)4483worker2 # [ 34.737049] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 55 ms (attempt 2/5)4484worker2 # [ 34.792193] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 123 ms (attempt 3/5)4485worker2 # [ 34.915463] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 797 ms (attempt 4/5)4486worker2 # [ 35.717342] nix-grpc-daemon[788]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4487worker2 # [ 35.720142] nix-grpc-daemon[788]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4488worker2 # [ 35.738063] nix-grpc-daemon[788]: ts=2026-09-21T12:58:04Z level=debug event=path_info_miss path=lnpvxnhllsvr2ncl4v29x83d7qqqszn3-nixos-system-client-test error="error: path '/nix/store/lnpvxnhllsvr2ncl4v29x83d7qqqszn3-nixos-system-client-test' is required, but there is no substituter that can build it"4489worker2 # [ 35.748879] nix-grpc-daemon[788]: ts=2026-09-21T12:58:04Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:546064490worker2 # [ 35.754226] nix-grpc-daemon[788]: ts=2026-09-21T12:58:04Z level=debug event=path_info_miss path=lnpvxnhllsvr2ncl4v29x83d7qqqszn3-nixos-system-client-test error="error: path '/nix/store/lnpvxnhllsvr2ncl4v29x83d7qqqszn3-nixos-system-client-test' is required, but there is no substituter that can build it"4491client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4492client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.09 seconds)4493client: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4494client: (finished: must succeed: install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)4495worker2 # [ 36.172662] nix-grpc-daemon[788]: ts=2026-09-21T12:58:05Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:546064496worker2 # [ 36.184540] nix-grpc-daemon[788]: ts=2026-09-21T12:58:05Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:546064497worker2 # [ 36.189317] nix-grpc-daemon[788]: ts=2026-09-21T12:58:05Z level=debug event=path_info_miss path=lnpvxnhllsvr2ncl4v29x83d7qqqszn3-nixos-system-client-test error="error: path '/nix/store/lnpvxnhllsvr2ncl4v29x83d7qqqszn3-nixos-system-client-test' is required, but there is no substituter that can build it"4498worker2 # [ 36.192924] nix-grpc-daemon[788]: ts=2026-09-21T12:58:05Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:546064499worker2 # [ 36.197717] nix-grpc-daemon[788]: ts=2026-09-21T12:58:05Z level=debug event=path_info_miss path=lnpvxnhllsvr2ncl4v29x83d7qqqszn3-nixos-system-client-test error="error: path '/nix/store/lnpvxnhllsvr2ncl4v29x83d7qqqszn3-nixos-system-client-test' is required, but there is no substituter that can build it"4500client: must succeed: rm -r /var/lib/nix-grpc-store4501client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.04 seconds)4502lb # [ 38.352145] envoy[922]: conn peer=[2001:db8:1::1]:60130 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=406 ms=22554503lb # [ 38.355834] envoy[922]: conn peer=[2001:db8:1::1]:60136 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=362 ms=344504worker1 # [ 38.474890] nix-grpc-daemon[784]: ts=2026-09-21T12:58:07Z level=info event=denied method=QueryMissing cn=- role=none4505worker2 # [ 38.798650] nix-grpc-daemon[788]: ts=2026-09-21T12:58:07Z level=info event=denied method=QueryMissing cn=stranger role=none4506(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 5.67 seconds)4507subtest: DAG build is scheduled across workers and lands in the cache4508client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24509worker2 # [ 39.044204] nix-grpc-daemon[788]: ts=2026-09-21T12:58:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:546064510client # don't know how to build these paths:4511client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4512worker2 # [ 39.058552] nix-grpc-daemon[788]: ts=2026-09-21T12:58:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:546064513worker2 # [ 39.096773] nix-grpc-daemon[788]: ts=2026-09-21T12:58:08Z level=debug event=want client=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv cached=04514worker2 # [ 39.101075] nix-grpc-daemon[788]: ts=2026-09-21T12:58:08Z level=debug event=assigned drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv worker=192.168.1.3:50051 clients=14515worker1 # [ 39.128318] nix-grpc-daemon[784]: ts=2026-09-21T12:58:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544224516worker1 # [ 39.156511] nix-grpc-daemon[784]: ts=2026-09-21T12:58:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544224517client # copying 1 paths to 'lb:50051'...4518worker1 # [ 39.167799] nix-grpc-daemon[784]: ts=2026-09-21T12:58:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544224519client # copying path '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' to 'lb:50051'...4520worker1 # [ 39.195308] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4521worker1 # [ 39.221818] nix-grpc-daemon[784]: ts=2026-09-21T12:58:08Z level=info event=niks3_push_started pid=8624522worker1 # [ 39.525196] nix-grpc-daemon[862]: time=2026-09-21T12:58:08.364Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""4523worker1 # [ 39.721227] nix-daemon[849]: accepted connection from pid 875, user nix-grpc-daemon (trusted)4524worker1 # [ 39.736931] nix-daemon[849]: reaped child process 878, status = succeeded4525worker1 # [ 39.913449] nix-daemon[849]: accepted connection from pid 881, user nix-grpc-daemon (trusted)4526worker1 # [ 39.925927] nix-daemon[849]: reaped child process 884, status = succeeded4527lb # [ 39.940406] niks3-server[1069]: 2026/09/21 12:58:08 INFO Received uploads request method=POST path=/api/pending_closures4528worker1 # [ 39.970102] nix-grpc-daemon[862]: time=2026-09-21T12:58:08.813Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4529worker1 # [ 39.972641] nix-grpc-daemon[862]: time=2026-09-21T12:58:08.816Z level=INFO msg="Uploading 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv (504B)"4530lb # [ 40.020241] niks3-server[1069]: 2026/09/21 12:58:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4531lb # [ 40.025249] niks3-server[1069]: 2026/09/21 12:58:08 INFO Registered completed upload object_key=nar/15vlbxqk65jh0v1qgc8308zz4diydc8wp113i0cc4ylv0zc5w964.nar.zst4532lb # [ 40.044116] niks3-server[1069]: 2026/09/21 12:58:08 INFO Signed narinfos id=1 count=14533worker1 # [ 40.049115] nix-grpc-daemon[862]: time=2026-09-21T12:58:08.892Z level=INFO msg="Uploading 1 narinfos"4534lb # [ 40.051320] niks3-server[1069]: 2026/09/21 12:58:08 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.ls4535lb # [ 40.057656] niks3-server[1069]: 2026/09/21 12:58:08 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4536lb # [ 40.067039] niks3-server[1069]: 2026/09/21 12:58:08 INFO Registered completed upload object_key=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1.narinfo4537lb # [ 40.072834] niks3-server[1069]: 2026/09/21 12:58:08 INFO Completed upload id=14538worker1 # [ 40.076412] nix-grpc-daemon[862]: time=2026-09-21T12:58:08.919Z level=INFO msg="Upload complete. (526ms)"4539worker1 # [ 40.081630] nix-grpc-daemon[784]: ts=2026-09-21T12:58:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=0 paths=1 nar_bytes_in=5044540worker1 # [ 40.088739] nix-grpc-daemon[784]: ts=2026-09-21T12:58:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544224541worker1 # [ 40.100296] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4542worker1 # [ 40.108650] nix-daemon[849]: reaped child process 860, status = succeeded4543client # building '/nix/store/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv' on lb:50051...4544worker1 # [ 40.117436] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4545client # farm-a> node-a: building 3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4546client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4547client # farm-a> LOG-farm-a-t14548worker1 # [ 40.941564] nix-daemon[849]: reaped child process 888, status = succeeded4549worker1 # [ 41.085146] nix-daemon[849]: accepted connection from pid 904, user nix-grpc-daemon (trusted)4550worker1 # [ 41.097176] nix-daemon[849]: reaped child process 907, status = succeeded4551lb # [ 41.112805] niks3-server[1069]: 2026/09/21 12:58:10 INFO Received uploads request method=POST path=/api/pending_closures4552worker1 # [ 41.127594] nix-grpc-daemon[862]: time=2026-09-21T12:58:09.970Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4553worker1 # [ 41.129263] nix-grpc-daemon[862]: time=2026-09-21T12:58:09.972Z level=INFO msg="Uploading qm96yd1cmricgf2hr21k88syh75l317v-farm-a (128B)"4554lb # [ 41.157678] niks3-server[1069]: 2026/09/21 12:58:10 INFO Registered completed upload object_key=log/3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv4555lb # [ 41.170255] niks3-server[1069]: 2026/09/21 12:58:10 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4556lb # [ 41.181209] niks3-server[1069]: 2026/09/21 12:58:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4557lb # [ 41.184582] niks3-server[1069]: 2026/09/21 12:58:10 INFO Signed narinfos id=2 count=14558worker1 # [ 41.191050] nix-grpc-daemon[862]: time=2026-09-21T12:58:10.033Z level=INFO msg="Uploading 1 narinfos"4559lb # [ 41.188948] niks3-server[1069]: 2026/09/21 12:58:10 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.ls4560lb # [ 41.198696] niks3-server[1069]: 2026/09/21 12:58:10 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4561lb # [ 41.203870] niks3-server[1069]: 2026/09/21 12:58:10 INFO Registered completed upload object_key=qm96yd1cmricgf2hr21k88syh75l317v.narinfo4562lb # [ 41.207094] niks3-server[1069]: 2026/09/21 12:58:10 INFO Completed upload id=24563worker1 # [ 41.210518] nix-grpc-daemon[862]: time=2026-09-21T12:58:10.053Z level=INFO msg="Upload complete. (282ms)"4564worker1 # [ 41.216974] nix-grpc-daemon[784]: ts=2026-09-21T12:58:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=1 drv=3ijpb6qq7zgbnifk6lc6i1fipl0iqkl1-farm-a.drv assign_id=1 outputs=14565worker1 # [ 41.229894] nix-daemon[849]: reaped child process 886, status = succeeded4566worker2 # [ 41.261806] nix-grpc-daemon[788]: ts=2026-09-21T12:58:10Z level=debug event=want client=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv cached=04567worker2 # [ 41.266184] nix-grpc-daemon[788]: ts=2026-09-21T12:58:10Z level=debug event=assigned drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv worker=192.168.1.3:50051 clients=14568worker1 # [ 41.286384] nix-grpc-daemon[784]: ts=2026-09-21T12:58:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544224569worker1 # [ 41.299134] nix-grpc-daemon[784]: ts=2026-09-21T12:58:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544224570worker1 # [ 41.313203] nix-grpc-daemon[862]: time=2026-09-21T12:58:10.156Z level=INFO msg="All 1 paths already cached"4571client # copying 1 paths to 'lb:50051'...4572worker1 # [ 41.321650] nix-grpc-daemon[784]: ts=2026-09-21T12:58:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544224573client # copying path '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' to 'lb:50051'...4574worker1 # [ 41.342175] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4575worker1 # [ 41.551566] nix-daemon[849]: accepted connection from pid 911, user nix-grpc-daemon (trusted)4576worker1 # [ 41.566175] nix-daemon[849]: reaped child process 914, status = succeeded4577worker1 # [ 41.752427] nix-daemon[849]: accepted connection from pid 916, user nix-grpc-daemon (trusted)4578worker1 # [ 41.765269] nix-daemon[849]: reaped child process 919, status = succeeded4579lb # [ 41.779444] niks3-server[1069]: 2026/09/21 12:58:10 INFO Received uploads request method=POST path=/api/pending_closures4580worker1 # [ 41.795976] nix-grpc-daemon[862]: time=2026-09-21T12:58:10.638Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4581worker1 # [ 41.797525] nix-grpc-daemon[862]: time=2026-09-21T12:58:10.638Z level=INFO msg="Uploading zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv (624B)"4582lb # [ 41.816342] niks3-server[1069]: 2026/09/21 12:58:10 INFO Registered completed upload object_key=nar/19sdpvl15ladfjss8gp73f6binhg0wgk0323wmn91mihvj530pxq.nar.zst4583lb # [ 41.821542] niks3-server[1069]: 2026/09/21 12:58:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4584worker1 # [ 41.832479] nix-grpc-daemon[862]: time=2026-09-21T12:58:10.675Z level=INFO msg="Uploading 1 narinfos"4585lb # [ 41.829126] niks3-server[1069]: 2026/09/21 12:58:10 INFO Signed narinfos id=3 count=14586lb # [ 41.835224] niks3-server[1069]: 2026/09/21 12:58:10 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.ls4587lb # [ 41.845350] niks3-server[1069]: 2026/09/21 12:58:10 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4588lb # [ 41.851899] niks3-server[1069]: 2026/09/21 12:58:10 INFO Completed upload id=34589worker1 # [ 41.855428] nix-grpc-daemon[862]: time=2026-09-21T12:58:10.698Z level=INFO msg="Upload complete. (499ms)"4590lb # [ 41.853551] niks3-server[1069]: 2026/09/21 12:58:10 INFO Registered completed upload object_key=zbhcql1mkb05ig74hjj18iqpvqy3rfbp.narinfo4591worker1 # [ 41.859186] nix-grpc-daemon[784]: ts=2026-09-21T12:58:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=0 paths=1 nar_bytes_in=6244592worker1 # [ 41.863849] nix-grpc-daemon[784]: ts=2026-09-21T12:58:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544224593worker1 # [ 41.875135] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4594worker1 # [ 41.879254] nix-daemon[849]: reaped child process 909, status = succeeded4595worker1 # [ 41.884452] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4596client # building '/nix/store/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv' on lb:50051...4597client # farm-b> node-a: building zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4598client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4599client # farm-b> LOG-farm-b-t14600worker1 # [ 43.298169] nix-daemon[849]: reaped child process 923, status = succeeded4601worker1 # [ 43.463345] nix-daemon[849]: accepted connection from pid 940, user nix-grpc-daemon (trusted)4602worker1 # [ 43.477364] nix-daemon[849]: reaped child process 943, status = succeeded4603lb # [ 43.490616] niks3-server[1069]: 2026/09/21 12:58:12 INFO Received uploads request method=POST path=/api/pending_closures4604worker1 # [ 43.502857] nix-grpc-daemon[862]: time=2026-09-21T12:58:12.346Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4605worker1 # [ 43.504421] nix-grpc-daemon[862]: time=2026-09-21T12:58:12.347Z level=INFO msg="Uploading jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b (176B)"4606lb # [ 43.523342] niks3-server[1069]: 2026/09/21 12:58:12 INFO Registered completed upload object_key=log/zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv4607lb # [ 43.536677] niks3-server[1069]: 2026/09/21 12:58:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4608lb # [ 43.542065] niks3-server[1069]: 2026/09/21 12:58:12 INFO Signed narinfos id=4 count=14609worker1 # [ 43.548548] nix-grpc-daemon[862]: time=2026-09-21T12:58:12.391Z level=INFO msg="Uploading 1 narinfos"4610lb # [ 43.550062] niks3-server[1069]: 2026/09/21 12:58:12 INFO Registered completed upload object_key=nar/0n7l6rw4y681bymb8bwra2nh13s7ybxcm2l7y8iwy0k5mmnaznjg.nar.zst4611lb # [ 43.557866] niks3-server[1069]: 2026/09/21 12:58:12 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.ls4612lb # [ 43.565501] niks3-server[1069]: 2026/09/21 12:58:12 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4613lb # [ 43.571534] niks3-server[1069]: 2026/09/21 12:58:12 INFO Completed upload id=44614lb # [ 43.572787] niks3-server[1069]: 2026/09/21 12:58:12 INFO Registered completed upload object_key=jicn175r0p4806ghnhxdvn7fsa1akkr0.narinfo4615worker1 # [ 43.575064] nix-grpc-daemon[862]: time=2026-09-21T12:58:12.418Z level=INFO msg="Upload complete. (281ms)"4616worker1 # [ 43.580654] nix-grpc-daemon[784]: ts=2026-09-21T12:58:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=1 drv=zbhcql1mkb05ig74hjj18iqpvqy3rfbp-farm-b.drv assign_id=2 outputs=14617worker2 # [ 43.613552] nix-grpc-daemon[788]: ts=2026-09-21T12:58:12Z level=debug event=want client=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=04618worker2 # [ 43.617543] nix-grpc-daemon[788]: ts=2026-09-21T12:58:12Z level=debug event=assigned drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv worker=192.168.1.3:50051 clients=14619worker1 # [ 43.596107] nix-daemon[849]: reaped child process 921, status = succeeded4620worker1 # [ 43.631612] nix-grpc-daemon[784]: ts=2026-09-21T12:58:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544224621worker1 # [ 43.644936] nix-grpc-daemon[784]: ts=2026-09-21T12:58:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544224622worker1 # [ 43.658402] nix-grpc-daemon[862]: time=2026-09-21T12:58:12.501Z level=INFO msg="All 2 paths already cached"4623client # copying 1 paths to 'lb:50051'...4624worker1 # [ 43.666955] nix-grpc-daemon[784]: ts=2026-09-21T12:58:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544224625client # copying path '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' to 'lb:50051'...4626worker1 # [ 43.687177] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4627worker1 # [ 43.885160] nix-daemon[849]: accepted connection from pid 947, user nix-grpc-daemon (trusted)4628worker1 # [ 43.899704] nix-daemon[849]: reaped child process 950, status = succeeded4629worker1 # [ 44.079065] nix-daemon[849]: accepted connection from pid 952, user nix-grpc-daemon (trusted)4630worker1 # [ 44.093414] nix-daemon[849]: reaped child process 955, status = succeeded4631lb # [ 44.105762] niks3-server[1069]: 2026/09/21 12:58:13 INFO Received uploads request method=POST path=/api/pending_closures4632worker1 # [ 44.118181] nix-grpc-daemon[862]: time=2026-09-21T12:58:12.961Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4633worker1 # [ 44.119537] nix-grpc-daemon[862]: time=2026-09-21T12:58:12.961Z level=INFO msg="Uploading c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv (752B)"4634lb # [ 44.135623] niks3-server[1069]: 2026/09/21 12:58:13 INFO Registered completed upload object_key=nar/1nf01qbgzkrsqfjd6d2bx60xxkfkbrljmfrvypvjci1c69ad7knd.nar.zst4635lb # [ 44.143403] niks3-server[1069]: 2026/09/21 12:58:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4636lb # [ 44.148484] niks3-server[1069]: 2026/09/21 12:58:13 INFO Signed narinfos id=5 count=14637worker1 # [ 44.154896] nix-grpc-daemon[862]: time=2026-09-21T12:58:12.998Z level=INFO msg="Uploading 1 narinfos"4638lb # [ 44.154978] niks3-server[1069]: 2026/09/21 12:58:13 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.ls4639lb # [ 44.168572] niks3-server[1069]: 2026/09/21 12:58:13 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4640lb # [ 44.173802] niks3-server[1069]: 2026/09/21 12:58:13 INFO Completed upload id=54641worker1 # [ 44.177340] nix-grpc-daemon[862]: time=2026-09-21T12:58:13.020Z level=INFO msg="Upload complete. (476ms)"4642lb # [ 44.176961] niks3-server[1069]: 2026/09/21 12:58:13 INFO Registered completed upload object_key=c7ysj984j3hnalhcvn7z0328xf5fr0zd.narinfo4643worker1 # [ 44.181484] nix-grpc-daemon[784]: ts=2026-09-21T12:58:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=0 paths=1 nar_bytes_in=7524644worker1 # [ 44.187212] nix-grpc-daemon[784]: ts=2026-09-21T12:58:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544224645worker1 # [ 44.195986] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4646worker1 # [ 44.201647] nix-daemon[849]: reaped child process 945, status = succeeded4647worker1 # [ 44.206973] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4648client # building '/nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv' on lb:50051...4649client # farm-top> node-a: building c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4650client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4651client # farm-top> LOG-farm-top-t14652worker1 # [ 45.019991] nix-daemon[849]: reaped child process 959, status = succeeded4653worker1 # [ 45.180859] nix-daemon[849]: accepted connection from pid 975, user nix-grpc-daemon (trusted)4654worker1 # [ 45.195404] nix-daemon[849]: reaped child process 978, status = succeeded4655lb # [ 45.208825] niks3-server[1069]: 2026/09/21 12:58:14 INFO Received uploads request method=POST path=/api/pending_closures4656worker1 # [ 45.222205] nix-grpc-daemon[862]: time=2026-09-21T12:58:14.065Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4657worker1 # [ 45.225293] nix-grpc-daemon[862]: time=2026-09-21T12:58:14.067Z level=INFO msg="Uploading 5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top (232B)"4658lb # [ 45.248782] niks3-server[1069]: 2026/09/21 12:58:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4659worker1 # [ 45.257059] nix-grpc-daemon[862]: time=2026-09-21T12:58:14.100Z level=INFO msg="Uploading 1 narinfos"4660lb # [ 45.253603] niks3-server[1069]: 2026/09/21 12:58:14 INFO Signed narinfos id=6 count=14661lb # [ 45.260360] niks3-server[1069]: 2026/09/21 12:58:14 INFO Registered completed upload object_key=nar/0pq8hp2qdfznpnf4pn4my9c4nd9yx6cfjn0x0bgv1rpk0536kpjy.nar.zst4662lb # [ 45.268410] niks3-server[1069]: 2026/09/21 12:58:14 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.ls4663lb # [ 45.271634] niks3-server[1069]: 2026/09/21 12:58:14 INFO Registered completed upload object_key=log/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4664lb # [ 45.275466] niks3-server[1069]: 2026/09/21 12:58:14 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4665lb # [ 45.279316] niks3-server[1069]: 2026/09/21 12:58:14 INFO Completed upload id=64666worker1 # [ 45.282792] nix-grpc-daemon[862]: time=2026-09-21T12:58:14.126Z level=INFO msg="Upload complete. (269ms)"4667lb # [ 45.280808] niks3-server[1069]: 2026/09/21 12:58:14 INFO Registered completed upload object_key=5qgbfb7awggfq3smlyfxgz8zixnqxk2n.narinfo4668worker1 # [ 45.288249] nix-grpc-daemon[784]: ts=2026-09-21T12:58:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=1 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv assign_id=3 outputs=14669worker1 # [ 45.297471] nix-daemon[849]: reaped child process 957, status = succeeded4670client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 6.53 seconds)4671client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4672client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.19 seconds)4673client: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4674client: (finished: must fail: test -e /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.02 seconds)4675client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4676client # copying 3 paths...4677client # copying path '/nix/store/qm96yd1cmricgf2hr21k88syh75l317v-farm-a' from 'http://lb:5751'...4678client # copying path '/nix/store/jicn175r0p4806ghnhxdvn7fsa1akkr0-farm-b' from 'http://lb:5751'...4679client # copying path '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top' from 'http://lb:5751'...4680client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top && grep farm-top-t1 /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.28 seconds)4681(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.12 seconds)4682subtest: repeat is answered Cached by the scheduler without building4683worker1: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4684worker1 # finding garbage collector roots...4685worker1 # deleting specified paths...4686worker1 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4687worker1 # deleting unused links...4688worker1 # note: hard linking is currently saving 0.0 KiB4689worker1: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.28 seconds)4690worker2: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top4691worker2 # finding garbage collector roots...4692worker2 # deleting specified paths...4693worker2 # deleting '/nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top'4694worker2 # deleting unused links...4695worker2 # note: hard linking is currently saving 0.0 KiB4696worker2: (finished: must succeed: nix-store --delete /nix/store/5qgbfb7awggfq3smlyfxgz8zixnqxk2n-farm-top, in 0.28 seconds)4697client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&24698client # don't know how to build these paths:4699client # /nix/store/c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv4700worker2 # [ 46.740654] nix-grpc-daemon[788]: ts=2026-09-21T12:58:15Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:546064701worker1 # [ 46.721422] nix-grpc-daemon[784]: ts=2026-09-21T12:58:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544224702worker1 # [ 46.727290] nix-grpc-daemon[862]: time=2026-09-21T12:58:15.570Z level=INFO msg="All 2 paths already cached"4703worker2 # [ 46.767874] nix-grpc-daemon[788]: ts=2026-09-21T12:58:15Z level=debug event=want client=2 drv=c7ysj984j3hnalhcvn7z0328xf5fr0zd-farm-top.drv cached=14704client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 >&2, in 0.30 seconds)4705client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath4706client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 outPath, in 0.17 seconds)4707lb: must succeed: curl -sf localhost:9901/clusters4708lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4709lb: must succeed: curl -sf localhost:9901/clusters4710lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4711worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4712worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4713(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.30 seconds)4714subtest: inputs already in the cache are not sent to the scheduler4715lb: must succeed: curl -sf localhost:9901/clusters4716lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4717lb: must succeed: curl -sf localhost:9901/clusters4718lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4719worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4720worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4721client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&24722worker1 # [ 47.610625] nix-grpc-daemon[784]: ts=2026-09-21T12:58:16Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:544224723client # don't know how to build these paths:4724client # /nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4725worker1 # [ 47.619245] nix-grpc-daemon[784]: ts=2026-09-21T12:58:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544224726worker1 # [ 47.627840] nix-grpc-daemon[862]: time=2026-09-21T12:58:16.471Z level=INFO msg="All 2 paths already cached"4727worker2 # [ 47.666737] nix-grpc-daemon[788]: ts=2026-09-21T12:58:16Z level=debug event=want client=3 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv cached=04728worker2 # [ 47.671215] nix-grpc-daemon[788]: ts=2026-09-21T12:58:16Z level=debug event=assigned drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv worker=192.168.1.3:50051 clients=14729worker1 # [ 47.685703] nix-grpc-daemon[784]: ts=2026-09-21T12:58:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544224730worker1 # [ 47.698672] nix-grpc-daemon[784]: ts=2026-09-21T12:58:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544224731worker1 # [ 47.711571] nix-grpc-daemon[862]: time=2026-09-21T12:58:16.554Z level=INFO msg="All 2 paths already cached"4732client # copying 1 paths to 'lb:50051'...4733client # copying path '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' to 'lb:50051'...4734worker1 # [ 47.720611] nix-grpc-daemon[784]: ts=2026-09-21T12:58:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544224735worker1 # [ 47.732877] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4736worker1 # [ 47.924178] nix-daemon[849]: accepted connection from pid 995, user nix-grpc-daemon (trusted)4737worker1 # [ 47.938352] nix-daemon[849]: reaped child process 998, status = succeeded4738worker1 # [ 48.113125] nix-daemon[849]: accepted connection from pid 1000, user nix-grpc-daemon (trusted)4739worker1 # [ 48.126803] nix-daemon[849]: reaped child process 1003, status = succeeded4740lb # [ 48.140743] niks3-server[1069]: 2026/09/21 12:58:17 INFO Received uploads request method=POST path=/api/pending_closures4741worker1 # [ 48.151807] nix-grpc-daemon[862]: time=2026-09-21T12:58:16.994Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4742worker1 # [ 48.153368] nix-grpc-daemon[862]: time=2026-09-21T12:58:16.996Z level=INFO msg="Uploading y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv (752B)"4743lb # [ 48.169614] niks3-server[1069]: 2026/09/21 12:58:17 INFO Registered completed upload object_key=nar/0p14b74h5654i3y6qv7zaaib0ya421b01knaq0m00r5ypjyh2q1d.nar.zst4744lb # [ 48.179596] niks3-server[1069]: 2026/09/21 12:58:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4745lb # [ 48.185081] niks3-server[1069]: 2026/09/21 12:58:17 INFO Signed narinfos id=7 count=14746worker1 # [ 48.191439] nix-grpc-daemon[862]: time=2026-09-21T12:58:17.034Z level=INFO msg="Uploading 1 narinfos"4747lb # [ 48.191554] niks3-server[1069]: 2026/09/21 12:58:17 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.ls4748lb # [ 48.202095] niks3-server[1069]: 2026/09/21 12:58:17 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4749worker1 # [ 48.212935] nix-grpc-daemon[862]: time=2026-09-21T12:58:17.056Z level=INFO msg="Upload complete. (467ms)"4750lb # [ 48.209775] niks3-server[1069]: 2026/09/21 12:58:17 INFO Completed upload id=74751lb # [ 48.213078] niks3-server[1069]: 2026/09/21 12:58:17 INFO Registered completed upload object_key=y2wm8aizyyg449ia2pcscrjc2pklsrz4.narinfo4752worker1 # [ 48.215778] nix-grpc-daemon[784]: ts=2026-09-21T12:58:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=0 paths=1 nar_bytes_in=7524753worker1 # [ 48.220946] nix-grpc-daemon[784]: ts=2026-09-21T12:58:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544224754worker1 # [ 48.229577] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4755worker1 # [ 48.235451] nix-daemon[849]: reaped child process 993, status = succeeded4756worker1 # [ 48.240932] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4757client # building '/nix/store/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv' on lb:50051...4758client # farm-top> node-a: building y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4759lb # [ 48.353504] envoy[922]: conn peer=[2001:db8:1::1]:60138 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=174760lb # [ 48.357070] envoy[922]: conn peer=[2001:db8:1::1]:60148 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=94761lb # [ 48.360562] envoy[922]: conn peer=[2001:db8:1::1]:60160 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9481 tx=4049 ms=63094762lb # [ 48.364092] envoy[922]: conn peer=[2001:db8:1::1]:35802 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=991 tx=480 ms=414763client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4764worker1 # [ 48.705103] nix-grpc-daemon[784]: E0000 00:00:1789995497.547182 824 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4765worker1 # [ 48.718426] nix-grpc-daemon[784]: E0000 00:00:1789995497.561760 824 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4766worker1 # [ 48.722479] nix-grpc-daemon[784]: E0000 00:00:1789995497.564042 824 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4767worker1 # [ 48.734768] nix-grpc-daemon[784]: E0000 00:00:1789995497.578150 824 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4768client # farm-top> LOG-farm-top-t1b4769worker1 # [ 48.863577] nix-daemon[849]: reaped child process 1007, status = succeeded4770worker1 # [ 49.029095] nix-daemon[849]: accepted connection from pid 1023, user nix-grpc-daemon (trusted)4771worker1 # [ 49.043760] nix-daemon[849]: reaped child process 1026, status = succeeded4772lb # [ 49.056718] niks3-server[1069]: 2026/09/21 12:58:17 INFO Received uploads request method=POST path=/api/pending_closures4773worker1 # [ 49.067707] nix-grpc-daemon[862]: time=2026-09-21T12:58:17.911Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4774worker1 # [ 49.069266] nix-grpc-daemon[862]: time=2026-09-21T12:58:17.912Z level=INFO msg="Uploading v688vnrkfcqiwaygym5ifmnr3yawh4xd-farm-top (232B)"4775lb # [ 49.093766] niks3-server[1069]: 2026/09/21 12:58:18 INFO Registered completed upload object_key=log/y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv4776lb # [ 49.108271] niks3-server[1069]: 2026/09/21 12:58:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4777lb # [ 49.113879] niks3-server[1069]: 2026/09/21 12:58:18 INFO Registered completed upload object_key=nar/1j36mvvk44wddjhqvk302zl80d251m1g0d4kc4gfqy00xyklwzbf.nar.zst4778lb # [ 49.119100] niks3-server[1069]: 2026/09/21 12:58:18 INFO Signed narinfos id=8 count=14779worker1 # [ 49.125033] nix-grpc-daemon[862]: time=2026-09-21T12:58:17.968Z level=INFO msg="Uploading 1 narinfos"4780lb # [ 49.128701] niks3-server[1069]: 2026/09/21 12:58:18 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.ls4781lb # [ 49.134486] niks3-server[1069]: 2026/09/21 12:58:18 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4782lb # [ 49.145299] niks3-server[1069]: 2026/09/21 12:58:18 INFO Completed upload id=84783worker1 # [ 49.148257] nix-grpc-daemon[862]: time=2026-09-21T12:58:17.991Z level=INFO msg="Upload complete. (300ms)"4784lb # [ 49.150139] niks3-server[1069]: 2026/09/21 12:58:18 INFO Registered completed upload object_key=v688vnrkfcqiwaygym5ifmnr3yawh4xd.narinfo4785worker1 # [ 49.152896] nix-grpc-daemon[784]: ts=2026-09-21T12:58:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=0 drv=y2wm8aizyyg449ia2pcscrjc2pklsrz4-farm-top.drv assign_id=4 outputs=14786worker1 # [ 49.167616] nix-daemon[849]: reaped child process 1005, status = succeeded4787client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b >&2, in 1.81 seconds)4788client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath4789client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.18 seconds)4790lb: must succeed: curl -sf localhost:9901/clusters4791lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4792lb: must succeed: curl -sf localhost:9901/clusters4793lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4794worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4795worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.05 seconds)4796(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.32 seconds)4797subtest: two clients wanting the same drv build it once4798client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4799client # [ 49.607144] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4800client # Running as unit: dedup1.service; invocation ID: 493ba7c8955847449fe87f04f95c205f4801client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 0.07 seconds)4802client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup4803client # [ 49.795473] nix[892]: don't know how to build these paths:4804client # [ 49.796400] nix[892]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4805worker1 # [ 49.778962] nix-grpc-daemon[784]: ts=2026-09-21T12:58:18Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:544224806worker2 # [ 49.817673] nix-grpc-daemon[788]: ts=2026-09-21T12:58:18Z level=debug event=want client=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04807worker2 # [ 49.822193] nix-grpc-daemon[788]: ts=2026-09-21T12:58:18Z level=debug event=assigned drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv worker=192.168.1.4:50051 clients=14808worker2 # [ 49.827785] nix-grpc-daemon[788]: ts=2026-09-21T12:58:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:546064809worker2 # [ 49.838650] nix-grpc-daemon[788]: ts=2026-09-21T12:58:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:546064810client # [ 49.837077] nix[892]: copying 1 paths to 'lb:50051'...4811client # [ 49.838789] nix[892]: copying path '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' to 'lb:50051'...4812worker2 # [ 49.849357] nix-grpc-daemon[788]: ts=2026-09-21T12:58:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:546064813worker2 # [ 49.860895] nix-daemon[873]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4814worker2 # [ 49.879506] nix-grpc-daemon[788]: ts=2026-09-21T12:58:18Z level=info event=niks3_push_started pid=9264815worker2 # [ 50.091971] nix-grpc-daemon[926]: time=2026-09-21T12:58:19.152Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""4816worker2 # [ 50.250356] nix-daemon[873]: accepted connection from pid 938, user nix-grpc-daemon (trusted)4817worker2 # [ 50.264502] nix-daemon[873]: reaped child process 941, status = succeeded4818worker2 # [ 50.397368] nix-daemon[873]: accepted connection from pid 943, user nix-grpc-daemon (trusted)4819worker2 # [ 50.409932] nix-daemon[873]: reaped child process 946, status = succeeded4820lb # [ 50.391898] niks3-server[1069]: 2026/09/21 12:58:19 INFO Received uploads request method=POST path=/api/pending_closures4821worker2 # [ 50.432266] nix-grpc-daemon[926]: time=2026-09-21T12:58:19.495Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4822worker2 # [ 50.434671] nix-grpc-daemon[926]: time=2026-09-21T12:58:19.498Z level=INFO msg="Uploading k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv (432B)"4823lb # [ 50.429783] niks3-server[1069]: 2026/09/21 12:58:19 INFO Registered completed upload object_key=nar/1rprin71wk1rn18ghpyxh5nm1akxfkjivwxfnlzqzmri5xdvl2l0.nar.zst4824lb # [ 50.437208] niks3-server[1069]: 2026/09/21 12:58:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4825worker2 # [ 50.471772] nix-grpc-daemon[926]: time=2026-09-21T12:58:19.535Z level=INFO msg="Uploading 1 narinfos"4826lb # [ 50.442180] niks3-server[1069]: 2026/09/21 12:58:19 INFO Signed narinfos id=9 count=14827lb # [ 50.452553] niks3-server[1069]: 2026/09/21 12:58:19 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.ls4828lb # [ 50.459922] niks3-server[1069]: 2026/09/21 12:58:19 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4829worker2 # [ 50.495880] nix-grpc-daemon[926]: time=2026-09-21T12:58:19.559Z level=INFO msg="Upload complete. (391ms)"4830lb # [ 50.466487] niks3-server[1069]: 2026/09/21 12:58:19 INFO Completed upload id=94831worker2 # [ 50.497848] nix-grpc-daemon[788]: ts=2026-09-21T12:58:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54606 duration_s=0 paths=1 nar_bytes_in=4324832worker2 # [ 50.501721] nix-grpc-daemon[788]: ts=2026-09-21T12:58:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:546064833lb # [ 50.471931] niks3-server[1069]: 2026/09/21 12:58:19 INFO Registered completed upload object_key=k75hvg0drf5n5cgm1w4hms82xb1l5nmz.narinfo4834worker2 # [ 50.508387] nix-daemon[873]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4835worker2 # [ 50.509632] nix-daemon[873]: reaped child process 924, status = succeeded4836worker2 # [ 50.514854] nix-daemon[873]: accepted connection from pid 788, user nix-grpc-daemon (trusted)4837client # [ 50.501973] nix[892]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4838client # [ 50.680280] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup.4839client # Running as unit: dedup2.service; invocation ID: fda0610185bd46bb9356c4fc27912bc64840client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag dedup, in 1.08 seconds)4841worker1 # [ 50.910711] nix-grpc-daemon[784]: ts=2026-09-21T12:58:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:544224842client # [ 50.930580] nix[908]: don't know how to build these paths:4843client # [ 50.931532] nix[908]: /nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4844worker2 # [ 50.950283] nix-grpc-daemon[788]: ts=2026-09-21T12:58:20Z level=debug event=want client=5 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv cached=04845worker2 # [ 50.959081] nix-grpc-daemon[788]: ts=2026-09-21T12:58:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:546064846client # [ 50.951986] nix[908]: building '/nix/store/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv' on lb:50051...4847client: must succeed: sleep 24848client: (finished: must succeed: sleep 2, in 2.05 seconds)4849client: waiting for success: ! systemctl is-active dedup1 dedup24850worker2 # [ 54.342880] nix-daemon[873]: reaped child process 950, status = succeeded4851worker2 # [ 54.482340] nix-daemon[873]: accepted connection from pid 983, user nix-grpc-daemon (trusted)4852worker2 # [ 54.493766] nix-daemon[873]: reaped child process 986, status = succeeded4853lb # [ 54.476754] niks3-server[1069]: 2026/09/21 12:58:23 INFO Received uploads request method=POST path=/api/pending_closures4854worker2 # [ 54.514280] nix-grpc-daemon[926]: time=2026-09-21T12:58:23.577Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4855worker2 # [ 54.515877] nix-grpc-daemon[926]: time=2026-09-21T12:58:23.579Z level=INFO msg="Uploading pihyvyr6gd3qnm368ii42wvcpgsfv1di-slow-dedup (120B)"4856lb # [ 54.521738] niks3-server[1069]: 2026/09/21 12:58:23 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4857lb # [ 54.528164] niks3-server[1069]: 2026/09/21 12:58:23 INFO Registered completed upload object_key=log/k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv4858lb # [ 54.536791] niks3-server[1069]: 2026/09/21 12:58:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4859worker2 # [ 54.572738] nix-grpc-daemon[926]: time=2026-09-21T12:58:23.635Z level=INFO msg="Uploading 1 narinfos"4860lb # [ 54.542830] niks3-server[1069]: 2026/09/21 12:58:23 INFO Signed narinfos id=10 count=14861lb # [ 54.546583] niks3-server[1069]: 2026/09/21 12:58:23 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.ls4862lb # [ 54.556088] niks3-server[1069]: 2026/09/21 12:58:23 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4863lb # [ 54.562239] niks3-server[1069]: 2026/09/21 12:58:23 INFO Completed upload id=104864worker2 # [ 54.591499] nix-grpc-daemon[926]: time=2026-09-21T12:58:23.654Z level=INFO msg="Upload complete. (262ms)"4865lb # [ 54.564991] niks3-server[1069]: 2026/09/21 12:58:23 INFO Registered completed upload object_key=pihyvyr6gd3qnm368ii42wvcpgsfv1di.narinfo4866worker2 # [ 54.598351] nix-grpc-daemon[788]: ts=2026-09-21T12:58:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54606 duration_s=3 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14867worker2 # [ 54.604243] nix-grpc-daemon[788]: ts=2026-09-21T12:58:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54606 duration_s=4 drv=k75hvg0drf5n5cgm1w4hms82xb1l5nmz-slow-dedup.drv assign_id=5 outputs=14868worker2 # [ 54.613686] nix-daemon[873]: reaped child process 948, status = succeeded4869client # [ 54.624482] systemd[1]: dedup2.service: Deactivated successfully.4870client # [ 54.628763] systemd[1]: dedup2.service: Consumed 138ms CPU time over 3.947s wall clock time, 35.1M memory peak, 5.9K incoming IP traffic, 5.6K outgoing IP traffic.4871client # [ 54.668960] systemd[1]: dedup1.service: Deactivated successfully.4872client # [ 54.670379] systemd[1]: dedup1.service: Consumed 119ms CPU time over 5.064s wall clock time, 41.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.4873client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.11 seconds)4874client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4875client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.04 seconds)4876worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4877worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.06 seconds)4878worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4879worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)4880worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4881worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4882worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4883worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4884(finished: subtest: two clients wanting the same drv build it once, in 5.98 seconds)4885subtest: build hook: nix-daemon with builders = grpc://lb4886client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log4887client # [ 55.737522] systemd[1]: Started Nix Daemon.4888client # [ 55.915029] nix-daemon[950]: accepted connection from pid 947, user root (trusted)4889worker1 # [ 56.098612] nix-grpc-daemon[784]: ts=2026-09-21T12:58:24Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:544224890worker1 # [ 56.103926] nix-grpc-daemon[784]: ts=2026-09-21T12:58:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544224891worker2 # [ 56.141398] nix-grpc-daemon[788]: ts=2026-09-21T12:58:25Z level=debug event=want client=6 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv cached=04892worker2 # [ 56.145559] nix-grpc-daemon[788]: ts=2026-09-21T12:58:25Z level=debug event=assigned drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv worker=192.168.1.3:50051 clients=14893worker1 # [ 56.158991] nix-grpc-daemon[784]: ts=2026-09-21T12:58:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544224894worker1 # [ 56.171208] nix-grpc-daemon[784]: ts=2026-09-21T12:58:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544224895worker1 # [ 56.183428] nix-grpc-daemon[784]: ts=2026-09-21T12:58:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544224896worker1 # [ 56.195337] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4897worker1 # [ 56.401934] nix-daemon[849]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)4898worker1 # [ 56.417272] nix-daemon[849]: reaped child process 1054, status = succeeded4899worker1 # [ 56.607266] nix-daemon[849]: accepted connection from pid 1056, user nix-grpc-daemon (trusted)4900worker1 # [ 56.621169] nix-daemon[849]: reaped child process 1059, status = succeeded4901lb # [ 56.636354] niks3-server[1069]: 2026/09/21 12:58:25 INFO Received uploads request method=POST path=/api/pending_closures4902worker1 # [ 56.647741] nix-grpc-daemon[862]: time=2026-09-21T12:58:25.490Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4903worker1 # [ 56.649276] nix-grpc-daemon[862]: time=2026-09-21T12:58:25.492Z level=INFO msg="Uploading pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv (512B)"4904lb # [ 56.673253] niks3-server[1069]: 2026/09/21 12:58:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4905lb # [ 56.680379] niks3-server[1069]: 2026/09/21 12:58:25 INFO Registered completed upload object_key=nar/08mhnmxkz6k9gck9zisdh6jb3dqb53xqy0jak6cm9lpasdvr4sk7.nar.zst4906worker1 # [ 56.690431] nix-grpc-daemon[862]: time=2026-09-21T12:58:25.533Z level=INFO msg="Uploading 1 narinfos"4907lb # [ 56.686769] niks3-server[1069]: 2026/09/21 12:58:25 INFO Signed narinfos id=11 count=14908lb # [ 56.694583] niks3-server[1069]: 2026/09/21 12:58:25 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.ls4909lb # [ 56.704105] niks3-server[1069]: 2026/09/21 12:58:25 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4910lb # [ 56.714637] niks3-server[1069]: 2026/09/21 12:58:25 INFO Completed upload id=114911worker1 # [ 56.717855] nix-grpc-daemon[862]: time=2026-09-21T12:58:25.561Z level=INFO msg="Upload complete. (510ms)"4912lb # [ 56.717433] niks3-server[1069]: 2026/09/21 12:58:25 INFO Registered completed upload object_key=pd51i21khb2xrd9paajf80pb96sw8zn0.narinfo4913worker1 # [ 56.721887] nix-grpc-daemon[784]: ts=2026-09-21T12:58:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=0 paths=1 nar_bytes_in=5124914worker1 # [ 56.728345] nix-grpc-daemon[784]: ts=2026-09-21T12:58:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544224915worker1 # [ 56.739110] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4916worker1 # [ 56.752544] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4917worker1 # [ 56.754727] nix-daemon[849]: reaped child process 1049, status = succeeded4918worker1 # [ 57.568521] nix-daemon[849]: reaped child process 1063, status = succeeded4919worker1 # [ 57.731691] nix-daemon[849]: accepted connection from pid 1079, user nix-grpc-daemon (trusted)4920worker1 # [ 57.745468] nix-daemon[849]: reaped child process 1082, status = succeeded4921lb # [ 57.760299] niks3-server[1069]: 2026/09/21 12:58:26 INFO Received uploads request method=POST path=/api/pending_closures4922worker1 # [ 57.772043] nix-grpc-daemon[862]: time=2026-09-21T12:58:26.615Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4923worker1 # [ 57.775062] nix-grpc-daemon[862]: time=2026-09-21T12:58:26.617Z level=INFO msg="Uploading 6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a (128B)"4924lb # [ 57.802432] niks3-server[1069]: 2026/09/21 12:58:26 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4925lb # [ 57.814580] niks3-server[1069]: 2026/09/21 12:58:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4926lb # [ 57.820417] niks3-server[1069]: 2026/09/21 12:58:26 INFO Signed narinfos id=12 count=14927worker1 # [ 57.827040] nix-grpc-daemon[862]: time=2026-09-21T12:58:26.670Z level=INFO msg="Uploading 1 narinfos"4928lb # [ 57.828286] niks3-server[1069]: 2026/09/21 12:58:26 INFO Registered completed upload object_key=log/pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv4929lb # [ 57.834674] niks3-server[1069]: 2026/09/21 12:58:26 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.ls4930lb # [ 57.843742] niks3-server[1069]: 2026/09/21 12:58:26 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4931lb # [ 57.848711] niks3-server[1069]: 2026/09/21 12:58:26 INFO Completed upload id=124932worker1 # [ 57.852183] nix-grpc-daemon[862]: time=2026-09-21T12:58:26.695Z level=INFO msg="Upload complete. (292ms)"4933lb # [ 57.850997] niks3-server[1069]: 2026/09/21 12:58:26 INFO Registered completed upload object_key=6zgzdy628dcdg0dbc5qi7nisk08n4kp2.narinfo4934worker1 # [ 57.857595] nix-grpc-daemon[784]: ts=2026-09-21T12:58:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=1 drv=pd51i21khb2xrd9paajf80pb96sw8zn0-farm-a.drv assign_id=6 outputs=14935worker1 # [ 57.865502] nix-grpc-daemon[784]: ts=2026-09-21T12:58:26Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:544224936worker1 # [ 57.875767] nix-daemon[849]: reaped child process 1061, status = succeeded4937worker1 # [ 57.878707] nix-grpc-daemon[784]: ts=2026-09-21T12:58:26Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:544224938worker1 # [ 57.883546] nix-grpc-daemon[784]: ts=2026-09-21T12:58:26Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=0 paths=1 nar_bytes_out=1284939worker1 # [ 58.159249] nix-grpc-daemon[784]: ts=2026-09-21T12:58:27Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:544224940worker1 # [ 58.165126] nix-grpc-daemon[784]: ts=2026-09-21T12:58:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544224941worker1 # [ 58.170088] nix-grpc-daemon[862]: time=2026-09-21T12:58:27.012Z level=INFO msg="All 1 paths already cached"4942worker2 # [ 58.207194] nix-grpc-daemon[788]: ts=2026-09-21T12:58:27Z level=debug event=want client=7 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv cached=04943worker1 # [ 58.183217] nix-grpc-daemon[784]: ts=2026-09-21T12:58:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544224944worker2 # [ 58.211343] nix-grpc-daemon[788]: ts=2026-09-21T12:58:27Z level=debug event=assigned drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv worker=192.168.1.3:50051 clients=14945worker1 # [ 58.195101] nix-grpc-daemon[784]: ts=2026-09-21T12:58:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544224946worker1 # [ 58.206980] nix-grpc-daemon[862]: time=2026-09-21T12:58:27.050Z level=INFO msg="All 1 paths already cached"4947worker1 # [ 58.213468] nix-grpc-daemon[784]: ts=2026-09-21T12:58:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544224948worker1 # [ 58.223162] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4949lb # [ 58.354177] envoy[922]: conn peer=[2001:db8:1::1]:35808 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4126 tx=1766 ms=15594950lb # [ 58.358060] envoy[922]: conn peer=[2001:db8:1::1]:35812 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1427 tx=1085 ms=36814951lb # [ 58.361574] envoy[922]: conn peer=[2001:db8:1::1]:35810 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2998 tx=1451 ms=48524952lb # [ 58.365150] envoy[922]: conn peer=[2001:db8:1::1]:51860 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3414 tx=1739 ms=18394953lb # [ 58.368630] envoy[922]: conn peer=[2001:db8:1::1]:51868 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=333 ms=574954worker1 # [ 58.418459] nix-daemon[849]: accepted connection from pid 1086, user nix-grpc-daemon (trusted)4955worker1 # [ 58.433780] nix-daemon[849]: reaped child process 1089, status = succeeded4956worker1 # [ 58.604685] nix-daemon[849]: accepted connection from pid 1091, user nix-grpc-daemon (trusted)4957worker1 # [ 58.618513] nix-daemon[849]: reaped child process 1094, status = succeeded4958lb # [ 58.630551] niks3-server[1069]: 2026/09/21 12:58:27 INFO Received uploads request method=POST path=/api/pending_closures4959worker1 # [ 58.641399] nix-grpc-daemon[862]: time=2026-09-21T12:58:27.484Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4960worker1 # [ 58.642909] nix-grpc-daemon[862]: time=2026-09-21T12:58:27.486Z level=INFO msg="Uploading vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv (624B)"4961lb # [ 58.657487] niks3-server[1069]: 2026/09/21 12:58:27 INFO Registered completed upload object_key=nar/1190412lxhybpj2b0886g97h0cpm7xx8r4virf8q4n7i91409ivg.nar.zst4962lb # [ 58.668771] niks3-server[1069]: 2026/09/21 12:58:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4963worker1 # [ 58.677124] nix-grpc-daemon[862]: time=2026-09-21T12:58:27.520Z level=INFO msg="Uploading 1 narinfos"4964lb # [ 58.673586] niks3-server[1069]: 2026/09/21 12:58:27 INFO Signed narinfos id=13 count=14965lb # [ 58.687561] niks3-server[1069]: 2026/09/21 12:58:27 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4966lb # [ 58.693229] niks3-server[1069]: 2026/09/21 12:58:27 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.ls4967lb # [ 58.697587] niks3-server[1069]: 2026/09/21 12:58:27 INFO Completed upload id=134968worker1 # [ 58.700988] nix-grpc-daemon[862]: time=2026-09-21T12:58:27.544Z level=INFO msg="Upload complete. (464ms)"4969lb # [ 58.700254] niks3-server[1069]: 2026/09/21 12:58:27 INFO Registered completed upload object_key=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8.narinfo4970worker1 # [ 58.704669] nix-grpc-daemon[784]: ts=2026-09-21T12:58:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=0 paths=1 nar_bytes_in=6244971worker1 # [ 58.710329] nix-grpc-daemon[784]: ts=2026-09-21T12:58:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544224972worker1 # [ 58.719694] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4973worker1 # [ 58.724557] nix-daemon[849]: reaped child process 1084, status = succeeded4974worker1 # [ 58.729252] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4975worker1 # [ 59.758365] nix-daemon[849]: reaped child process 1098, status = succeeded4976worker1 # [ 59.924716] nix-daemon[849]: accepted connection from pid 1114, user nix-grpc-daemon (trusted)4977worker1 # [ 59.938748] nix-daemon[849]: reaped child process 1117, status = succeeded4978lb # [ 59.952249] niks3-server[1069]: 2026/09/21 12:58:28 INFO Received uploads request method=POST path=/api/pending_closures4979worker1 # [ 59.963568] nix-grpc-daemon[862]: time=2026-09-21T12:58:28.806Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4980worker1 # [ 59.966307] nix-grpc-daemon[862]: time=2026-09-21T12:58:28.808Z level=INFO msg="Uploading mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b (176B)"4981lb # [ 59.988160] niks3-server[1069]: 2026/09/21 12:58:28 INFO Registered completed upload object_key=log/vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv4982lb # [ 59.994821] niks3-server[1069]: 2026/09/21 12:58:28 INFO Registered completed upload object_key=nar/1mgm44m4ndgi5l3yi2qv3v3sx48190m8d4ycw3r4pfzy4z1wq2s6.nar.zst4983lb # [ 60.006693] niks3-server[1069]: 2026/09/21 12:58:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4984lb # [ 60.012587] niks3-server[1069]: 2026/09/21 12:58:28 INFO Signed narinfos id=14 count=14985worker1 # [ 60.019398] nix-grpc-daemon[862]: time=2026-09-21T12:58:28.862Z level=INFO msg="Uploading 1 narinfos"4986lb # [ 60.015977] niks3-server[1069]: 2026/09/21 12:58:28 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.ls4987lb # [ 60.026269] niks3-server[1069]: 2026/09/21 12:58:28 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4988lb # [ 60.031250] niks3-server[1069]: 2026/09/21 12:58:28 INFO Completed upload id=144989worker1 # [ 60.034724] nix-grpc-daemon[862]: time=2026-09-21T12:58:28.878Z level=INFO msg="Upload complete. (297ms)"4990lb # [ 60.034396] niks3-server[1069]: 2026/09/21 12:58:28 INFO Registered completed upload object_key=mrrcl9jb4hh6rly2df2vfym5sq1235hf.narinfo4991worker1 # [ 60.041592] nix-grpc-daemon[784]: ts=2026-09-21T12:58:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=1 drv=vwnd00v6q8nvv6nb52vdjqfl5vdlzdd8-farm-b.drv assign_id=7 outputs=14992worker2 # [ 60.072064] nix-grpc-daemon[788]: ts=2026-09-21T12:58:29Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:546064993worker1 # [ 60.054364] nix-daemon[849]: reaped child process 1096, status = succeeded4994worker2 # [ 60.087076] nix-grpc-daemon[788]: copying path '/nix/store/6zgzdy628dcdg0dbc5qi7nisk08n4kp2-farm-a' from 'http://lb:5751'...4995worker2 # [ 60.102297] nix-grpc-daemon[788]: copying path '/nix/store/mrrcl9jb4hh6rly2df2vfym5sq1235hf-farm-b' from 'http://lb:5751'...4996worker1 # [ 60.104154] nix-grpc-daemon[784]: ts=2026-09-21T12:58:28Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:544224997worker1 # [ 60.108755] nix-grpc-daemon[784]: ts=2026-09-21T12:58:28Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=0 paths=1 nar_bytes_out=1764998worker1 # [ 60.324315] nix-grpc-daemon[784]: ts=2026-09-21T12:58:29Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:544224999worker2 # [ 60.357718] nix-grpc-daemon[788]: ts=2026-09-21T12:58:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:546065000worker2 # [ 60.367584] nix-grpc-daemon[926]: time=2026-09-21T12:58:29.430Z level=INFO msg="All 2 paths already cached"5001worker2 # [ 60.381226] nix-grpc-daemon[788]: ts=2026-09-21T12:58:29Z level=debug event=want client=8 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv cached=05002worker2 # [ 60.385608] nix-grpc-daemon[788]: ts=2026-09-21T12:58:29Z level=debug event=assigned drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv worker=192.168.1.3:50051 clients=15003worker1 # [ 60.361848] nix-grpc-daemon[784]: ts=2026-09-21T12:58:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544225004worker1 # [ 60.373828] nix-grpc-daemon[784]: ts=2026-09-21T12:58:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544225005worker1 # [ 60.385696] nix-grpc-daemon[862]: time=2026-09-21T12:58:29.229Z level=INFO msg="All 2 paths already cached"5006worker1 # [ 60.393374] nix-grpc-daemon[784]: ts=2026-09-21T12:58:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544225007worker1 # [ 60.401734] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5008worker1 # [ 60.582530] nix-daemon[849]: accepted connection from pid 1121, user nix-grpc-daemon (trusted)5009worker1 # [ 60.597119] nix-daemon[849]: reaped child process 1124, status = succeeded5010worker1 # [ 60.759863] nix-daemon[849]: accepted connection from pid 1126, user nix-grpc-daemon (trusted)5011worker1 # [ 60.773603] nix-daemon[849]: reaped child process 1129, status = succeeded5012lb # [ 60.784760] niks3-server[1069]: 2026/09/21 12:58:29 INFO Received uploads request method=POST path=/api/pending_closures5013worker1 # [ 60.796313] nix-grpc-daemon[862]: time=2026-09-21T12:58:29.639Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5014worker1 # [ 60.797850] nix-grpc-daemon[862]: time=2026-09-21T12:58:29.641Z level=INFO msg="Uploading bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv (752B)"5015lb # [ 60.815494] niks3-server[1069]: 2026/09/21 12:58:29 INFO Registered completed upload object_key=nar/0d4r016fc570pq02p1j53gahxygvvyr62937d84sa58ri3rzk000.nar.zst5016lb # [ 60.826267] niks3-server[1069]: 2026/09/21 12:58:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign5017lb # [ 60.831816] niks3-server[1069]: 2026/09/21 12:58:29 INFO Signed narinfos id=15 count=15018worker1 # [ 60.838292] nix-grpc-daemon[862]: time=2026-09-21T12:58:29.681Z level=INFO msg="Uploading 1 narinfos"5019lb # [ 60.838314] niks3-server[1069]: 2026/09/21 12:58:29 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.ls5020lb # [ 60.849388] niks3-server[1069]: 2026/09/21 12:58:29 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete5021lb # [ 60.855096] niks3-server[1069]: 2026/09/21 12:58:29 INFO Completed upload id=155022worker1 # [ 60.858543] nix-grpc-daemon[862]: time=2026-09-21T12:58:29.701Z level=INFO msg="Upload complete. (442ms)"5023worker1 # [ 60.861967] nix-grpc-daemon[784]: ts=2026-09-21T12:58:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=0 paths=1 nar_bytes_in=7525024lb # [ 60.860093] niks3-server[1069]: 2026/09/21 12:58:29 INFO Registered completed upload object_key=bshr7sr3z72cf1fa1a1b8fifr45jgpsw.narinfo5025worker1 # [ 60.867624] nix-grpc-daemon[784]: ts=2026-09-21T12:58:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544225026worker1 # [ 60.875554] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5027worker1 # [ 60.882209] nix-daemon[849]: reaped child process 1119, status = succeeded5028worker1 # [ 60.886402] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5029worker1 # [ 62.310513] nix-daemon[849]: reaped child process 1133, status = succeeded5030worker1 # [ 62.463376] nix-daemon[849]: accepted connection from pid 1149, user nix-grpc-daemon (trusted)5031worker1 # [ 62.477782] nix-daemon[849]: reaped child process 1152, status = succeeded5032lb # [ 62.491412] niks3-server[1069]: 2026/09/21 12:58:31 INFO Received uploads request method=POST path=/api/pending_closures5033worker1 # [ 62.503881] nix-grpc-daemon[862]: time=2026-09-21T12:58:31.347Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5034worker1 # [ 62.505578] nix-grpc-daemon[862]: time=2026-09-21T12:58:31.348Z level=INFO msg="Uploading aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top (232B)"5035lb # [ 62.532072] niks3-server[1069]: 2026/09/21 12:58:31 INFO Registered completed upload object_key=nar/1r64sjcvz2vpjvwv3psvwf0bfifdzwiafvy1m89v5is1mm18ds87.nar.zst5036lb # [ 62.541339] niks3-server[1069]: 2026/09/21 12:58:31 INFO Registered completed upload object_key=log/bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv5037lb # [ 62.548253] niks3-server[1069]: 2026/09/21 12:58:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign5038lb # [ 62.552143] niks3-server[1069]: 2026/09/21 12:58:31 INFO Signed narinfos id=16 count=15039worker1 # [ 62.558247] nix-grpc-daemon[862]: time=2026-09-21T12:58:31.401Z level=INFO msg="Uploading 1 narinfos"5040lb # [ 62.559257] niks3-server[1069]: 2026/09/21 12:58:31 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.ls5041lb # [ 62.563728] niks3-server[1069]: 2026/09/21 12:58:31 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete5042lb # [ 62.568095] niks3-server[1069]: 2026/09/21 12:58:31 INFO Completed upload id=165043worker1 # [ 62.571512] nix-grpc-daemon[862]: time=2026-09-21T12:58:31.414Z level=INFO msg="Upload complete. (277ms)"5044lb # [ 62.570410] niks3-server[1069]: 2026/09/21 12:58:31 INFO Registered completed upload object_key=aayfavl0hksy8qs9qc49znrwnfc5d2sr.narinfo5045worker1 # [ 62.575762] nix-grpc-daemon[784]: ts=2026-09-21T12:58:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=1 drv=bshr7sr3z72cf1fa1a1b8fifr45jgpsw-farm-top.drv assign_id=8 outputs=15046worker2 # [ 62.607438] nix-grpc-daemon[788]: ts=2026-09-21T12:58:31Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:546065047worker1 # [ 62.587957] nix-daemon[849]: reaped child process 1131, status = succeeded5048worker2 # [ 62.616597] nix-grpc-daemon[788]: copying path '/nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top' from 'http://lb:5751'...5049worker1 # [ 62.614169] nix-grpc-daemon[784]: ts=2026-09-21T12:58:31Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:544225050worker1 # [ 62.618407] nix-grpc-daemon[784]: ts=2026-09-21T12:58:31Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=0 paths=1 nar_bytes_out=2325051client # [ 62.738522] nix-daemon[950]: reaped child process 952, status = succeeded5052client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --print-out-paths --no-link -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.21 seconds)5053client: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top5054client: (finished: must succeed: grep farm-top-hook /nix/store/aayfavl0hksy8qs9qc49znrwnfc5d2sr-farm-top, in 0.03 seconds)5055client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }5056client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.03 seconds)5057client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }5058client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)5059(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.30 seconds)5060subtest: an input only one worker has still reaches the builder5061client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only5062client # this derivation will be built:5063client # /nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv5064client # building '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv'...5065client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag w1only, in 0.50 seconds)5066client: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure5067client: (finished: must succeed: nix-store --export /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only > /tmp/shared/inp.closure, in 0.19 seconds)5068worker1: must succeed: nix-store --import < /tmp/shared/inp.closure5069worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.23 seconds)5070worker2: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only5071worker2: (finished: must fail: test -e /nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only, in 0.03 seconds)5072client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&25073client # [ 63.999929] nix-daemon[950]: accepted connection from pid 1033, user root (trusted)5074client # this derivation will be built:5075client # /nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5076client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5077worker1 # [ 64.189888] nix-grpc-daemon[784]: ts=2026-09-21T12:58:33Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:544225078worker2 # [ 64.221072] nix-grpc-daemon[788]: ts=2026-09-21T12:58:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:546065079client # copying 1 paths...5080client # copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' to 'grpc://lb:50051'...5081worker1 # [ 64.206102] nix-grpc-daemon[784]: ts=2026-09-21T12:58:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544225082worker1 # [ 64.217336] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5083worker1 # [ 64.409475] nix-daemon[849]: accepted connection from pid 1161, user nix-grpc-daemon (trusted)5084worker1 # [ 64.423908] nix-daemon[849]: reaped child process 1164, status = succeeded5085lb # [ 64.437506] niks3-server[1069]: 2026/09/21 12:58:33 INFO Received uploads request method=POST path=/api/pending_closures5086worker1 # [ 64.448821] nix-grpc-daemon[862]: time=2026-09-21T12:58:33.291Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5087worker1 # [ 64.450683] nix-grpc-daemon[862]: time=2026-09-21T12:58:33.293Z level=INFO msg="Uploading gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only (120B)"5088lb # [ 64.463805] niks3-server[1069]: 2026/09/21 12:58:33 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst5089lb # [ 64.476061] niks3-server[1069]: 2026/09/21 12:58:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign5090worker1 # [ 64.484339] nix-grpc-daemon[862]: time=2026-09-21T12:58:33.327Z level=INFO msg="Uploading 1 narinfos"5091lb # [ 64.481141] niks3-server[1069]: 2026/09/21 12:58:33 INFO Signed narinfos id=17 count=15092lb # [ 64.487207] niks3-server[1069]: 2026/09/21 12:58:33 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.ls5093lb # [ 64.496818] niks3-server[1069]: 2026/09/21 12:58:33 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete5094worker1 # [ 64.506194] nix-grpc-daemon[862]: time=2026-09-21T12:58:33.349Z level=INFO msg="Upload complete. (279ms)"5095lb # [ 64.502890] niks3-server[1069]: 2026/09/21 12:58:33 INFO Completed upload id=175096worker1 # [ 64.508351] nix-grpc-daemon[784]: ts=2026-09-21T12:58:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=0 paths=1 nar_bytes_in=1205097lb # [ 64.507902] niks3-server[1069]: 2026/09/21 12:58:33 INFO Registered completed upload object_key=gycxa7jajm34fqsmi3xn0zjr3pvaya0x.narinfo5098worker1 # [ 64.517981] nix-daemon[849]: reaped child process 1159, status = succeeded5099worker2 # [ 64.549604] nix-grpc-daemon[788]: ts=2026-09-21T12:58:33Z level=debug event=want client=9 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv cached=05100worker2 # [ 64.555161] nix-grpc-daemon[788]: ts=2026-09-21T12:58:33Z level=debug event=assigned drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=15101worker2 # [ 64.593848] nix-grpc-daemon[788]: ts=2026-09-21T12:58:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:546065102worker2 # [ 64.605638] nix-grpc-daemon[788]: ts=2026-09-21T12:58:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:546065103client # copying 2 paths to 'lb:50051'...5104client # copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' to 'lb:50051'...5105worker2 # [ 64.622098] nix-grpc-daemon[788]: ts=2026-09-21T12:58:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:546065106client # copying path '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' to 'lb:50051'...5107worker2 # [ 64.636500] nix-daemon[873]: accepted connection from pid 788, user nix-grpc-daemon (trusted)5108worker2 # [ 64.832619] nix-daemon[873]: accepted connection from pid 1013, user nix-grpc-daemon (trusted)5109worker2 # [ 64.845528] nix-daemon[873]: reaped child process 1016, status = succeeded5110worker2 # [ 65.013688] nix-daemon[873]: accepted connection from pid 1018, user nix-grpc-daemon (trusted)5111worker2 # [ 65.023729] nix-daemon[873]: reaped child process 1021, status = succeeded5112lb # [ 65.010342] niks3-server[1069]: 2026/09/21 12:58:33 INFO Received uploads request method=POST path=/api/pending_closures5113lb # [ 65.019436] niks3-server[1069]: 2026/09/21 12:58:33 INFO Received uploads request method=POST path=/api/pending_closures5114worker2 # [ 65.057279] nix-grpc-daemon[926]: time=2026-09-21T12:58:34.120Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5115worker2 # [ 65.059377] nix-grpc-daemon[926]: time=2026-09-21T12:58:34.122Z level=INFO msg="Uploading k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv (560B)"5116worker2 # [ 65.061000] nix-grpc-daemon[926]: time=2026-09-21T12:58:34.122Z level=INFO msg="Uploading s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv (416B)"5117lb # [ 65.049475] niks3-server[1069]: 2026/09/21 12:58:33 INFO Registered completed upload object_key=nar/0s7awvvc5phwxzwwzhl47kzq38pdgwj6jp3kwxf5w2xddnxnkl6h.nar.zst5118lb # [ 65.063887] niks3-server[1069]: 2026/09/21 12:58:33 INFO Registered completed upload object_key=nar/136ms4k64sfvy4m17vvd9w9yncq4zydv4cjbhchdcg4j7ccbx3b9.nar.zst5119lb # [ 65.075124] niks3-server[1069]: 2026/09/21 12:58:33 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.ls5120lb # [ 65.082454] niks3-server[1069]: 2026/09/21 12:58:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign5121lb # [ 65.086154] niks3-server[1069]: 2026/09/21 12:58:33 INFO Signed narinfos id=18 count=15122lb # [ 65.088368] niks3-server[1069]: 2026/09/21 12:58:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign5123lb # [ 65.090379] niks3-server[1069]: 2026/09/21 12:58:34 INFO Signed narinfos id=19 count=25124lb # [ 65.091723] niks3-server[1069]: 2026/09/21 12:58:34 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.ls5125worker2 # [ 65.122916] nix-grpc-daemon[926]: time=2026-09-21T12:58:34.184Z level=INFO msg="Uploading 3 narinfos"5126lb # [ 65.105773] niks3-server[1069]: 2026/09/21 12:58:34 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5127lb # [ 65.107528] niks3-server[1069]: 2026/09/21 12:58:34 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete5128lb # [ 65.109646] niks3-server[1069]: 2026/09/21 12:58:34 INFO Registered completed upload object_key=s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr.narinfo5129lb # [ 65.112444] niks3-server[1069]: 2026/09/21 12:58:34 INFO Completed upload id=185130lb # [ 65.114772] niks3-server[1069]: 2026/09/21 12:58:34 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete5131lb # [ 65.116745] niks3-server[1069]: 2026/09/21 12:58:34 INFO Registered completed upload object_key=k9va03aiirjmgih65p90fpa9mr396d2b.narinfo5132lb # [ 65.118821] niks3-server[1069]: 2026/09/21 12:58:34 INFO Completed upload id=195133worker2 # [ 65.149645] nix-grpc-daemon[926]: time=2026-09-21T12:58:34.213Z level=INFO msg="Upload complete. (499ms)"5134worker2 # [ 65.153997] nix-grpc-daemon[788]: ts=2026-09-21T12:58:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54606 duration_s=0 paths=2 nar_bytes_in=9765135worker2 # [ 65.160592] nix-grpc-daemon[788]: ts=2026-09-21T12:58:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:546065136worker2 # [ 65.170937] nix-daemon[873]: accepted connection from pid 788, user nix-grpc-daemon (trusted)5137worker2 # [ 65.178495] nix-daemon[873]: reaped child process 1011, status = succeeded5138worker2 # [ 65.187771] nix-grpc-daemon[788]: copying path '/nix/store/gycxa7jajm34fqsmi3xn0zjr3pvaya0x-local-input-w1only' from 'http://lb:5751'...5139worker2 # [ 65.196377] nix-daemon[873]: accepted connection from pid 788, user nix-grpc-daemon (trusted)5140client # building '/nix/store/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv' on lb:50051...5141client # uses-input-w1onlya> worker2: building k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5142client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5143worker2 # [ 66.013987] nix-daemon[873]: reaped child process 1026, status = succeeded5144worker2 # [ 66.175736] nix-daemon[873]: accepted connection from pid 1042, user nix-grpc-daemon (trusted)5145worker2 # [ 66.187307] nix-daemon[873]: reaped child process 1045, status = succeeded5146lb # [ 66.174709] niks3-server[1069]: 2026/09/21 12:58:35 INFO Received uploads request method=POST path=/api/pending_closures5147worker2 # [ 66.214314] nix-grpc-daemon[926]: time=2026-09-21T12:58:35.277Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5148lb # [ 66.205350] niks3-server[1069]: 2026/09/21 12:58:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign5149lb # [ 66.209890] niks3-server[1069]: 2026/09/21 12:58:35 INFO Signed narinfos id=20 count=15150worker2 # [ 66.242160] nix-grpc-daemon[926]: time=2026-09-21T12:58:35.305Z level=INFO msg="Uploading 1 narinfos"5151lb # [ 66.220471] niks3-server[1069]: 2026/09/21 12:58:35 INFO Registered completed upload object_key=log/k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv5152lb # [ 66.226704] niks3-server[1069]: 2026/09/21 12:58:35 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.ls5153lb # [ 66.237574] niks3-server[1069]: 2026/09/21 12:58:35 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete5154worker2 # [ 66.273219] nix-grpc-daemon[926]: time=2026-09-21T12:58:35.336Z level=INFO msg="Upload complete. (269ms)"5155lb # [ 66.243805] niks3-server[1069]: 2026/09/21 12:58:35 INFO Completed upload id=205156lb # [ 66.245991] niks3-server[1069]: 2026/09/21 12:58:35 INFO Registered completed upload object_key=7fihpafywah8yif4mhbmz6i2i6adiqwj.narinfo5157worker2 # [ 66.277501] nix-grpc-daemon[788]: ts=2026-09-21T12:58:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54606 duration_s=1 drv=k9va03aiirjmgih65p90fpa9mr396d2b-uses-input-w1onlya.drv assign_id=9 outputs=15158client # copying 1 paths...5159worker2 # [ 66.287648] nix-grpc-daemon[788]: ts=2026-09-21T12:58:35Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:546065160client # copying path '/nix/store/7fihpafywah8yif4mhbmz6i2i6adiqwj-uses-input-w1onlya' from 'grpc://lb:50051'...5161worker2 # [ 66.294762] nix-daemon[873]: reaped child process 1023, status = succeeded5162worker2 # [ 66.305529] nix-grpc-daemon[788]: ts=2026-09-21T12:58:35Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:546065163worker2 # [ 66.307694] nix-grpc-daemon[788]: ts=2026-09-21T12:58:35Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:54606 duration_s=0 paths=1 nar_bytes_out=1205164client # [ 66.400364] nix-daemon[950]: reaped child process 1036, status = succeeded5165client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.63 seconds)5166client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&25167client # [ 66.586652] nix-daemon[950]: accepted connection from pid 1055, user root (trusted)5168client # this derivation will be built:5169client # /nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5170worker1 # [ 66.771868] nix-grpc-daemon[784]: ts=2026-09-21T12:58:35Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:544225171client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5172worker2 # [ 66.801163] nix-grpc-daemon[788]: ts=2026-09-21T12:58:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:546065173worker2 # [ 66.810570] nix-grpc-daemon[926]: time=2026-09-21T12:58:35.874Z level=INFO msg="All 1 paths already cached"5174worker2 # [ 66.824629] nix-grpc-daemon[788]: ts=2026-09-21T12:58:35Z level=debug event=want client=10 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv cached=05175worker1 # [ 66.801375] nix-grpc-daemon[784]: ts=2026-09-21T12:58:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544225176worker2 # [ 66.829491] nix-grpc-daemon[788]: ts=2026-09-21T12:58:35Z level=debug event=assigned drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=15177worker1 # [ 66.813914] nix-grpc-daemon[784]: ts=2026-09-21T12:58:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544225178client # copying 1 paths to 'lb:50051'...5179client # copying path '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' to 'lb:50051'...5180worker1 # [ 66.832930] nix-grpc-daemon[784]: ts=2026-09-21T12:58:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:544225181worker1 # [ 66.845211] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5182worker1 # [ 66.862309] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 14 ms (attempt 1/5)5183worker1 # [ 66.881119] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 8 ms (attempt 2/5)5184worker1 # [ 66.894422] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 224 ms (attempt 3/5)5185worker1 # [ 67.124402] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 735 ms (attempt 4/5)5186worker1 # [ 67.866694] nix-grpc-daemon[784]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5187worker1 # [ 67.871145] nix-grpc-daemon[784]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5188worker1 # [ 67.876560] nix-grpc-daemon[784]: copying path '/nix/store/s56ppgyv7qz2bls7ls2jsv2ni3g7fhhr-local-input-w1only.drv' from 'http://lb:5751'...5189worker1 # [ 68.072740] nix-daemon[849]: accepted connection from pid 1173, user nix-grpc-daemon (trusted)5190worker1 # [ 68.087081] nix-daemon[849]: reaped child process 1176, status = succeeded5191worker1 # [ 68.246206] nix-daemon[849]: accepted connection from pid 1178, user nix-grpc-daemon (trusted)5192worker1 # [ 68.259888] nix-daemon[849]: reaped child process 1181, status = succeeded5193lb # [ 68.272446] niks3-server[1069]: 2026/09/21 12:58:37 INFO Received uploads request method=POST path=/api/pending_closures5194worker1 # [ 68.284856] nix-grpc-daemon[862]: time=2026-09-21T12:58:37.128Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5195worker1 # [ 68.286443] nix-grpc-daemon[862]: time=2026-09-21T12:58:37.129Z level=INFO msg="Uploading 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv (560B)"5196lb # [ 68.305231] niks3-server[1069]: 2026/09/21 12:58:37 INFO Registered completed upload object_key=nar/0bp8pfsa12j91bab6f50rs7hh6sphjrig1zqlr7v80mn285qxax2.nar.zst5197lb # [ 68.312929] niks3-server[1069]: 2026/09/21 12:58:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign5198lb # [ 68.318579] niks3-server[1069]: 2026/09/21 12:58:37 INFO Signed narinfos id=21 count=15199worker1 # [ 68.324777] nix-grpc-daemon[862]: time=2026-09-21T12:58:37.168Z level=INFO msg="Uploading 1 narinfos"5200lb # [ 68.327312] niks3-server[1069]: 2026/09/21 12:58:37 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.ls5201lb # [ 68.336955] niks3-server[1069]: 2026/09/21 12:58:37 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete5202lb # [ 68.343048] niks3-server[1069]: 2026/09/21 12:58:37 INFO Completed upload id=215203worker1 # [ 68.346582] nix-grpc-daemon[862]: time=2026-09-21T12:58:37.189Z level=INFO msg="Upload complete. (447ms)"5204lb # [ 68.344539] niks3-server[1069]: 2026/09/21 12:58:37 INFO Registered completed upload object_key=2ign0w9nlrywl4qfkq5i314zinl1cxqs.narinfo5205worker1 # [ 68.350477] nix-grpc-daemon[784]: ts=2026-09-21T12:58:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=1 paths=1 nar_bytes_in=5605206lb # [ 68.353554] envoy[922]: conn peer=[2001:db8:1::1]:51872 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3893 tx=2069 ms=19625207worker1 # [ 68.355449] nix-grpc-daemon[784]: ts=2026-09-21T12:58:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:544225208lb # [ 68.354952] envoy[922]: conn peer=[2001:db8:1::1]:51876 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=409 tx=378 ms=185209lb # [ 68.356314] envoy[922]: conn peer=[2001:db8:1::1]:51888 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4326 tx=2301 ms=23415210lb # [ 68.357644] envoy[922]: conn peer=[2001:db8:1::1]:51900 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=437 tx=410 ms=515211lb # [ 68.359324] envoy[922]: conn peer=[2001:db8:1::1]:55628 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=545212lb # [ 68.360697] envoy[922]: conn peer=[2001:db8:1::1]:55624 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4197 tx=1827 ms=21465213worker1 # [ 68.364513] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5214worker1 # [ 68.373449] nix-daemon[849]: reaped child process 1166, status = succeeded5215client # building '/nix/store/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv' on lb:50051...5216worker1 # [ 68.376937] nix-daemon[849]: accepted connection from pid 784, user nix-grpc-daemon (trusted)5217client # uses-input-w1onlyb> node-a: building 2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5218client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5219worker1 # [ 69.405117] nix-daemon[849]: reaped child process 1185, status = succeeded5220worker1 # [ 69.571733] nix-daemon[849]: accepted connection from pid 1201, user nix-grpc-daemon (trusted)5221worker1 # [ 69.584622] nix-daemon[849]: reaped child process 1204, status = succeeded5222lb # [ 69.597520] niks3-server[1069]: 2026/09/21 12:58:38 INFO Received uploads request method=POST path=/api/pending_closures5223worker1 # [ 69.608746] nix-grpc-daemon[862]: time=2026-09-21T12:58:38.451Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5224lb # [ 69.628951] niks3-server[1069]: 2026/09/21 12:58:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign5225lb # [ 69.633413] niks3-server[1069]: 2026/09/21 12:58:38 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.ls5226worker1 # [ 69.644195] nix-grpc-daemon[862]: time=2026-09-21T12:58:38.487Z level=INFO msg="Uploading 1 narinfos"5227lb # [ 69.640914] niks3-server[1069]: 2026/09/21 12:58:38 INFO Signed narinfos id=22 count=15228lb # [ 69.645986] niks3-server[1069]: 2026/09/21 12:58:38 INFO Registered completed upload object_key=log/2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv5229lb # [ 69.658714] niks3-server[1069]: 2026/09/21 12:58:38 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete5230worker1 # [ 69.667861] nix-grpc-daemon[862]: time=2026-09-21T12:58:38.511Z level=INFO msg="Upload complete. (283ms)"5231lb # [ 69.664493] niks3-server[1069]: 2026/09/21 12:58:38 INFO Completed upload id=225232worker1 # [ 69.670704] nix-grpc-daemon[784]: ts=2026-09-21T12:58:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54422 duration_s=1 drv=2ign0w9nlrywl4qfkq5i314zinl1cxqs-uses-input-w1onlyb.drv assign_id=10 outputs=15233lb # [ 69.667238] niks3-server[1069]: 2026/09/21 12:58:38 INFO Registered completed upload object_key=rs93p5yqksm4zlblqlsfa8kxl8lkhrbg.narinfo5234client # copying 1 paths...5235worker1 # [ 69.678925] nix-daemon[849]: reaped child process 1183, status = succeeded5236worker2 # [ 69.707989] nix-grpc-daemon[788]: ts=2026-09-21T12:58:38Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:546065237worker2 # [ 69.717962] nix-grpc-daemon[788]: copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'http://lb:5751'...5238client # copying path '/nix/store/rs93p5yqksm4zlblqlsfa8kxl8lkhrbg-uses-input-w1onlyb' from 'grpc://lb:50051'...5239worker2 # [ 69.752167] nix-grpc-daemon[788]: ts=2026-09-21T12:58:38Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:546065240worker2 # [ 69.756456] nix-grpc-daemon[788]: ts=2026-09-21T12:58:38Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:54606 duration_s=0 paths=1 nar_bytes_out=1205241client # [ 69.842850] nix-daemon[950]: reaped child process 1058, status = succeeded5242client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.44 seconds)5243client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&25244client # [ 70.030984] nix-daemon[950]: accepted connection from pid 1077, user root (trusted)5245client # this derivation will be built:5246client # /nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5247worker2 # [ 70.247070] nix-grpc-daemon[788]: ts=2026-09-21T12:58:39Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:546065248client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux'...5249worker1 # [ 70.224118] nix-grpc-daemon[784]: ts=2026-09-21T12:58:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:544225250worker1 # [ 70.233302] nix-grpc-daemon[862]: time=2026-09-21T12:58:39.076Z level=INFO msg="All 1 paths already cached"5251worker2 # [ 70.273996] nix-grpc-daemon[788]: ts=2026-09-21T12:58:39Z level=debug event=want client=11 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv cached=05252worker2 # [ 70.280652] nix-grpc-daemon[788]: ts=2026-09-21T12:58:39Z level=debug event=assigned drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=15253worker2 # [ 70.285641] nix-grpc-daemon[788]: ts=2026-09-21T12:58:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:546065254worker2 # [ 70.296357] nix-grpc-daemon[788]: ts=2026-09-21T12:58:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:546065255worker2 # [ 70.309402] nix-grpc-daemon[926]: time=2026-09-21T12:58:39.372Z level=INFO msg="All 1 paths already cached"5256client # copying 1 paths to 'lb:50051'...5257client # copying path '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' to 'lb:50051'...5258worker2 # [ 70.318150] nix-grpc-daemon[788]: ts=2026-09-21T12:58:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:546065259worker2 # [ 70.329296] nix-daemon[873]: accepted connection from pid 788, user nix-grpc-daemon (trusted)5260worker2 # [ 70.530146] nix-daemon[873]: accepted connection from pid 1054, user nix-grpc-daemon (trusted)5261worker2 # [ 70.542451] nix-daemon[873]: reaped child process 1057, status = succeeded5262worker2 # [ 70.732846] nix-daemon[873]: accepted connection from pid 1059, user nix-grpc-daemon (trusted)5263worker2 # [ 70.744537] nix-daemon[873]: reaped child process 1062, status = succeeded5264lb # [ 70.732073] niks3-server[1069]: 2026/09/21 12:58:39 INFO Received uploads request method=POST path=/api/pending_closures5265worker2 # [ 70.771718] nix-grpc-daemon[926]: time=2026-09-21T12:58:39.834Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5266worker2 # [ 70.773163] nix-grpc-daemon[926]: time=2026-09-21T12:58:39.835Z level=INFO msg="Uploading ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv (560B)"5267lb # [ 70.759669] niks3-server[1069]: 2026/09/21 12:58:39 INFO Registered completed upload object_key=nar/1iq9qm6vw4b3p68mlkiw8nf0phmz21m8g79izsy14sydxmd4gmqa.nar.zst5268lb # [ 70.770909] niks3-server[1069]: 2026/09/21 12:58:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign5269lb # [ 70.774900] niks3-server[1069]: 2026/09/21 12:58:39 INFO Signed narinfos id=23 count=15270worker2 # [ 70.808364] nix-grpc-daemon[926]: time=2026-09-21T12:58:39.871Z level=INFO msg="Uploading 1 narinfos"5271lb # [ 70.781996] niks3-server[1069]: 2026/09/21 12:58:39 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.ls5272lb # [ 70.790929] niks3-server[1069]: 2026/09/21 12:58:39 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete5273worker2 # [ 70.827255] nix-grpc-daemon[926]: time=2026-09-21T12:58:39.890Z level=INFO msg="Upload complete. (486ms)"5274lb # [ 70.798086] niks3-server[1069]: 2026/09/21 12:58:39 INFO Completed upload id=235275lb # [ 70.801079] niks3-server[1069]: 2026/09/21 12:58:39 INFO Registered completed upload object_key=ych53lrh5qfmdxlnx672apx2v80x3aan.narinfo5276worker2 # [ 70.830233] nix-grpc-daemon[788]: ts=2026-09-21T12:58:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54606 duration_s=0 paths=1 nar_bytes_in=5605277worker2 # [ 70.835774] nix-grpc-daemon[788]: ts=2026-09-21T12:58:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:546065278worker2 # [ 70.845537] nix-daemon[873]: accepted connection from pid 788, user nix-grpc-daemon (trusted)5279worker2 # [ 70.848416] nix-daemon[873]: reaped child process 1052, status = succeeded5280worker2 # [ 70.853537] nix-daemon[873]: accepted connection from pid 788, user nix-grpc-daemon (trusted)5281client # building '/nix/store/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv' on lb:50051...5282client # uses-input-w1onlyc> worker2: building ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5283client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5284worker2 # [ 71.678607] nix-daemon[873]: reaped child process 1066, status = succeeded5285worker2 # [ 71.866339] nix-daemon[873]: accepted connection from pid 1082, user nix-grpc-daemon (trusted)5286worker2 # [ 71.877751] nix-daemon[873]: reaped child process 1085, status = succeeded5287lb # [ 71.867238] niks3-server[1069]: 2026/09/21 12:58:40 INFO Received uploads request method=POST path=/api/pending_closures5288worker2 # [ 71.903365] nix-grpc-daemon[926]: time=2026-09-21T12:58:40.966Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5289lb # [ 71.897115] niks3-server[1069]: 2026/09/21 12:58:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign5290lb # [ 71.900693] niks3-server[1069]: 2026/09/21 12:58:40 INFO Registered completed upload object_key=log/ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv5291lb # [ 71.905903] niks3-server[1069]: 2026/09/21 12:58:40 INFO Signed narinfos id=24 count=15292worker2 # [ 71.938254] nix-grpc-daemon[926]: time=2026-09-21T12:58:41.001Z level=INFO msg="Uploading 1 narinfos"5293lb # [ 71.914398] niks3-server[1069]: 2026/09/21 12:58:40 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.ls5294lb # [ 71.925443] niks3-server[1069]: 2026/09/21 12:58:40 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete5295lb # [ 71.931702] niks3-server[1069]: 2026/09/21 12:58:40 INFO Registered completed upload object_key=af2xkgzqdrcsfmk8dgmypxmxww9cx3rk.narinfo5296lb # [ 71.935604] niks3-server[1069]: 2026/09/21 12:58:40 INFO Completed upload id=245297worker2 # [ 71.965315] nix-grpc-daemon[926]: time=2026-09-21T12:58:41.028Z level=INFO msg="Upload complete. (304ms)"5298worker2 # [ 71.970646] nix-grpc-daemon[788]: ts=2026-09-21T12:58:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54606 duration_s=1 drv=ych53lrh5qfmdxlnx672apx2v80x3aan-uses-input-w1onlyc.drv assign_id=11 outputs=15299client # copying 1 paths...5300worker2 # [ 71.980602] nix-grpc-daemon[788]: ts=2026-09-21T12:58:41Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:546065301client # copying path '/nix/store/af2xkgzqdrcsfmk8dgmypxmxww9cx3rk-uses-input-w1onlyc' from 'grpc://lb:50051'...5302worker2 # [ 71.987208] nix-daemon[873]: reaped child process 1064, status = succeeded5303worker2 # [ 72.003110] nix-grpc-daemon[788]: ts=2026-09-21T12:58:41Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:546065304worker2 # [ 72.006537] nix-grpc-daemon[788]: ts=2026-09-21T12:58:41Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:54606 duration_s=0 paths=1 nar_bytes_out=1205305client # [ 72.099745] nix-daemon[950]: reaped child process 1080, status = succeeded5306client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&system=x86_64-linux x86_64-linux - 4' --no-link -f /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.26 seconds)5307(finished: subtest: an input only one worker has still reaches the builder, in 9.27 seconds)5308subtest: upload whose reference the worker lacks is completed from the cache5309client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache5310client # this derivation will be built:5311client # /nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv5312client # building '/nix/store/zpv7g9hl0xcjhjz0g0hd92sv1sf580zh-local-input-viacache.drv'...5313client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A input --argstr tag viacache, in 0.42 seconds)5314client: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache5315client # this derivation will be built:5316client # /nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv5317client # building '/nix/store/8higgmxp1c6ag4yj6sk9g6z66n43x70l-refers-to-input-viacache.drv'...5318client: (finished: must succeed: nix-build --no-out-link /nix/store/c7y5zg34h61m4kmsb9zg0afqx5zqqhrd-dep.nix -A referrer --argstr tag viacache, in 0.42 seconds)5319client: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure5320client: (finished: must succeed: nix-store --export /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache > /tmp/shared/ref.closure, in 0.20 seconds)5321worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5322worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.24 seconds)5323worker2 # [ 73.463708] systemd[1]: Stopping Nix worker-protocol over gRPC...5324worker2 # [ 73.737669] nix-grpc-daemon[788]: ts=2026-09-21T12:58:42Z level=info event=signal_exit5325lb # [ 73.963756] niks3-server[1069]: 2026/09/21 12:58:42 INFO lead: released remote=192.168.1.4:444805326worker2 # [ 74.289873] nix-grpc-daemon[788]: ts=2026-09-21T12:58:43Z level=info event=scheduler_restarting5327worker2 # [ 74.297749] nix-grpc-daemon[788]: ts=2026-09-21T12:58:43Z level=info event=scheduler_restarting addr=lb:500515328worker1 # [ 74.272622] nix-grpc-daemon[784]: ts=2026-09-21T12:58:43Z level=info event=scheduler_restarting addr=lb:500515329worker2 # [ 74.303173] nix-grpc-daemon[788]: ts=2026-09-21T12:58:43Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:441365330worker2 # [ 74.306986] nix-grpc-daemon[788]: ts=2026-09-21T12:58:43Z level=info event=worker_gone addr=192.168.1.4:500515331worker2 # [ 74.310816] nix-grpc-daemon[788]: ts=2026-09-21T12:58:43Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:441365332worker2 # [ 74.314293] nix-grpc-daemon[788]: ts=2026-09-21T12:58:43Z level=info event=worker_gone addr=192.168.1.3:500515333worker2 # [ 74.333350] nix-daemon[873]: reaped child process 875, status = succeeded5334worker1 # [ 74.474172] nix-grpc-daemon[784]: ts=2026-09-21T12:58:43Z level=info event=scheduler_connected addr=lb:500515335worker1 # [ 74.477952] nix-grpc-daemon[784]: ts=2026-09-21T12:58:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5336worker2 # [ 74.508438] nix-grpc-daemon[788]: ts=2026-09-21T12:58:43Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5337worker1 # [ 74.679540] nix-grpc-daemon[784]: ts=2026-09-21T12:58:43Z level=info event=scheduler_connected addr=lb:500515338worker1 # [ 74.683605] nix-grpc-daemon[784]: ts=2026-09-21T12:58:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5339worker2 # [ 74.848954] nix-grpc-daemon[788]: E0000 00:00:1789995523.911513 833 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5340worker2 # [ 74.889090] nix-grpc-daemon[788]: E0000 00:00:1789995523.952439 833 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5341worker1 # [ 74.884418] nix-grpc-daemon[784]: ts=2026-09-21T12:58:43Z level=info event=scheduler_connected addr=lb:500515342worker1 # [ 74.888369] nix-grpc-daemon[784]: ts=2026-09-21T12:58:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5343worker1 # [ 75.089249] nix-grpc-daemon[784]: ts=2026-09-21T12:58:43Z level=info event=scheduler_connected addr=lb:500515344worker1 # [ 75.093320] nix-grpc-daemon[784]: ts=2026-09-21T12:58:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5345worker1 # [ 75.295466] nix-grpc-daemon[784]: ts=2026-09-21T12:58:44Z level=info event=scheduler_connected addr=lb:500515346worker1 # [ 75.299155] nix-grpc-daemon[784]: ts=2026-09-21T12:58:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5347worker1 # [ 75.500213] nix-grpc-daemon[784]: ts=2026-09-21T12:58:44Z level=info event=scheduler_connected addr=lb:500515348worker1 # [ 75.504312] nix-grpc-daemon[784]: ts=2026-09-21T12:58:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5349worker2 # [ 75.612555] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5350worker2 # [ 75.616182] systemd[1]: Stopped Nix worker-protocol over gRPC.5351worker2 # [ 75.620852] systemd[1]: nix-grpc-daemon.service: Consumed 1.454s CPU time over 57.111s wall clock time, 52.9M memory peak, 54.9K incoming IP traffic, 175.1K outgoing IP traffic.5352worker2 # [ 75.632158] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5353worker2 # [ 75.635500] systemd[1]: Closed Nix worker-protocol over gRPC.5354worker2 # [ 75.638188] systemd[1]: nix-grpc-daemon.socket: Consumed 290us CPU time over 1min 6.256s wall clock time, 256K memory peak, 75.9K incoming IP traffic, 64.4K outgoing IP traffic.5355lb: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy5356lb: must succeed: curl -sf localhost:9901/clusters5357lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5358lb: (finished: waiting until exactly ['worker2'] endpoints of x86_64-linux are unhealthy, in 0.08 seconds)5359client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&25360worker1 # [ 75.705139] nix-grpc-daemon[784]: ts=2026-09-21T12:58:44Z level=info event=scheduler_connected addr=lb:500515361worker1 # [ 75.708997] nix-grpc-daemon[784]: ts=2026-09-21T12:58:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5362worker1 # [ 75.909830] nix-grpc-daemon[784]: ts=2026-09-21T12:58:44Z level=info event=scheduler_connected addr=lb:500515363worker1 # [ 75.913902] nix-grpc-daemon[784]: ts=2026-09-21T12:58:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5364worker1 # [ 75.953332] nix-grpc-daemon[784]: ts=2026-09-21T12:58:44Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:544225365client # /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5366worker1 # [ 75.958930] nix-grpc-daemon[784]: ts=2026-09-21T12:58:44Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:544225367client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache >&2, in 0.29 seconds)5368worker2: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache5369worker2: (finished: must fail: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache, in 0.03 seconds)5370worker2 # [ 76.102669] systemd[1]: Listening on Nix worker-protocol over gRPC.5371worker1 # [ 76.114907] nix-grpc-daemon[784]: ts=2026-09-21T12:58:44Z level=info event=scheduler_connected addr=lb:500515372worker1 # [ 76.118960] nix-grpc-daemon[784]: ts=2026-09-21T12:58:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5373worker1 # [ 76.149490] systemd[1]: Stopping Nix worker-protocol over gRPC...5374worker1 # [ 76.321450] nix-grpc-daemon[784]: ts=2026-09-21T12:58:45Z level=info event=scheduler_connected addr=lb:500515375worker1 # [ 76.324612] nix-grpc-daemon[784]: ts=2026-09-21T12:58:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5376lb # [ 76.494667] niks3-server[1069]: 2026/09/21 12:58:45 INFO lead: acquired remote=192.168.1.3:541305377worker1 # [ 76.501566] nix-grpc-daemon[784]: ts=2026-09-21T12:58:45Z level=info event=scheduler_take_over5378worker1 # [ 76.522547] nix-grpc-daemon[784]: ts=2026-09-21T12:58:45Z level=info event=scheduler_connected addr=lb:500515379worker1 # [ 76.526361] nix-grpc-daemon[784]: ts=2026-09-21T12:58:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5380worker2 # [ 76.597153] systemd[1]: Starting scheduler-token.service...5381worker2 # [ 76.704587] systemd[1]: scheduler-token.service: Deactivated successfully.5382worker2 # [ 76.707352] systemd[1]: Finished scheduler-token.service.5383worker2 # [ 76.709256] systemd[1]: scheduler-token.service: Consumed 35ms CPU time over 106ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5384worker2 # [ 76.715268] systemd[1]: Starting Nix worker-protocol over gRPC...5385worker1 # [ 76.692236] nix-grpc-daemon[784]: ts=2026-09-21T12:58:45Z level=info event=signal_exit5386worker1 # [ 76.727463] nix-grpc-daemon[784]: ts=2026-09-21T12:58:45Z level=info event=scheduler_connected addr=lb:500515387worker1 # [ 76.731433] nix-grpc-daemon[784]: ts=2026-09-21T12:58:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5388worker2 # [ 76.900592] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:45Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5389worker2 # [ 76.902361] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:45Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5390worker2 # [ 76.905995] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:45Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15391worker2 # [ 76.914066] nix-grpc-daemon[1109]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5392worker2 # [ 76.915512] nix-grpc-daemon[1109]: E0000 00:00:1789995525.976726 1109 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5393worker2 # [ 76.920843] nix-grpc-daemon[1109]: E0000 00:00:1789995525.981905 1109 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5394worker2 # [ 76.924278] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:45Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5395worker2 # [ 76.928312] systemd[1]: Started Nix worker-protocol over gRPC.5396worker2 # [ 76.936066] nix-grpc-daemon[1109]: E0000 00:00:1789995525.998696 1150 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5397worker2 # [ 76.947111] nix-grpc-daemon[1109]: E0000 00:00:1789995526.010588 1150 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5398worker2 # [ 76.950205] nix-grpc-daemon[1109]: E0000 00:00:1789995526.012714 1150 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5399worker1 # [ 76.932557] nix-grpc-daemon[784]: ts=2026-09-21T12:58:45Z level=info event=scheduler_connected addr=lb:500515400worker2 # [ 76.960944] nix-grpc-daemon[1109]: E0000 00:00:1789995526.024445 1150 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5401worker1 # [ 76.936564] nix-grpc-daemon[784]: ts=2026-09-21T12:58:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5402worker2 # [ 76.969787] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:46Z level=info event=scheduler_connected addr=lb:500515403worker2 # [ 76.971345] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5404lb # [ 76.958253] niks3-server[1069]: 2026/09/21 12:58:45 INFO lead: released remote=192.168.1.3:541305405worker1 # [ 77.137661] nix-grpc-daemon[784]: ts=2026-09-21T12:58:45Z level=info event=scheduler_connected addr=lb:500515406worker1 # [ 77.141635] nix-grpc-daemon[784]: ts=2026-09-21T12:58:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5407worker2 # [ 77.172814] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:46Z level=info event=scheduler_connected addr=lb:500515408worker2 # [ 77.176603] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5409worker1 # [ 77.260698] nix-grpc-daemon[784]: ts=2026-09-21T12:58:46Z level=info event=scheduler_restarting5410worker1 # [ 77.288500] nix-daemon[849]: reaped child process 851, status = succeeded5411worker2 # [ 77.377593] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:46Z level=info event=scheduler_connected addr=lb:500515412worker2 # [ 77.381487] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5413worker1 # [ 77.359408] nix-grpc-daemon[784]: ts=2026-09-21T12:58:46Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5414worker1 # [ 77.523861] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5415worker1 # [ 77.526973] systemd[1]: Stopped Nix worker-protocol over gRPC.5416worker1 # [ 77.532197] systemd[1]: nix-grpc-daemon.service: Consumed 2.909s CPU time over 59.177s wall clock time, 62.3M memory peak, 86.8K incoming IP traffic, 166.9K outgoing IP traffic.5417worker1 # [ 77.542723] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5418worker1 # [ 77.545964] systemd[1]: Closed Nix worker-protocol over gRPC.5419worker1 # [ 77.548301] systemd[1]: nix-grpc-daemon.socket: Consumed 298us CPU time over 1min 7.896s wall clock time, 256K memory peak, 73.5K incoming IP traffic, 60.7K outgoing IP traffic.5420lb: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy5421lb: must succeed: curl -sf localhost:9901/clusters5422worker2 # [ 77.582599] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:46Z level=info event=scheduler_connected addr=lb:500515423worker2 # [ 77.586506] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5424lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5425lb: (finished: waiting until exactly ['worker1'] endpoints of x86_64-linux are unhealthy, in 0.08 seconds)5426client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&25427worker2 # [ 77.787698] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:46Z level=info event=scheduler_connected addr=lb:500515428worker2 # [ 77.791532] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5429worker2 # [ 77.915786] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563705430worker2 # [ 77.927788] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)5431worker2 # [ 77.946954] nix-grpc-daemon[1109]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 23 ms (attempt 1/5)5432worker2 # [ 77.975785] nix-grpc-daemon[1109]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 70 ms (attempt 2/5)5433worker2 # [ 77.992334] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:47Z level=info event=scheduler_connected addr=lb:500515434worker2 # [ 77.996131] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5435worker2 # [ 78.051796] nix-grpc-daemon[1109]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 66 ms (attempt 3/5)5436worker2 # [ 78.123459] nix-grpc-daemon[1109]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 135 ms (attempt 4/5)5437worker2 # [ 78.197192] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:47Z level=info event=scheduler_connected addr=lb:500515438worker2 # [ 78.201253] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5439worker2 # [ 78.264721] nix-grpc-daemon[1109]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5440worker2 # [ 78.267555] nix-grpc-daemon[1109]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5441client # copying 2 paths...5442client # copying path '/nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache' to 'grpc://lb:50051'...5443worker2 # [ 78.297130] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563705444client # copying path '/nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache' to 'grpc://lb:50051'...5445worker2 # [ 78.309466] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)5446worker2 # [ 78.331360] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:47Z level=info event=niks3_push_started pid=11695447lb # [ 78.355525] envoy[922]: conn peer=[2001:db8:1::1]:55638 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=555448lb # [ 78.359338] envoy[922]: conn peer=[2001:db8:1::1]:55634 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3744 tx=2013 ms=30085449lb # [ 78.362806] envoy[922]: conn peer=[2001:db8:1::1]:55666 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=447 tx=328 ms=565450lb # [ 78.366916] envoy[922]: conn peer=[2001:db8:1::1]:55652 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3733 tx=1899 ms=18125451worker2 # [ 78.401748] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:47Z level=info event=scheduler_connected addr=lb:500515452lb # [ 78.370746] envoy[922]: conn peer=192.168.1.4:38410 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=54606 tx=2602 ms=557045453worker2 # [ 78.404887] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5454lb # [ 78.375345] envoy[922]: conn peer=[2001:db8:1::1]:36766 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=155455lb # [ 78.379652] envoy[922]: conn peer=192.168.1.3:38110 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=16245 tx=3244 ms=576195456worker2 # [ 78.571882] nix-grpc-daemon[1169]: time=2026-09-21T12:58:47.634Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""5457worker2 # [ 78.605497] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:47Z level=info event=scheduler_connected addr=lb:500515458worker2 # [ 78.607383] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5459worker2 # [ 78.776634] nix-daemon[873]: accepted connection from pid 1182, user nix-grpc-daemon (trusted)5460worker2 # [ 78.787074] nix-daemon[873]: reaped child process 1185, status = succeeded5461worker2 # [ 78.807843] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:47Z level=info event=scheduler_connected addr=lb:500515462worker2 # [ 78.809863] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5463lb # [ 78.781790] niks3-server[1069]: 2026/09/21 12:58:47 INFO Received uploads request method=POST path=/api/pending_closures5464lb # [ 78.791507] niks3-server[1069]: 2026/09/21 12:58:47 INFO Received uploads request method=POST path=/api/pending_closures5465worker2 # [ 78.828937] nix-grpc-daemon[1169]: time=2026-09-21T12:58:47.892Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5466worker2 # [ 78.830938] nix-grpc-daemon[1169]: time=2026-09-21T12:58:47.892Z level=INFO msg="Uploading cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache (128B)"5467worker2 # [ 78.832516] nix-grpc-daemon[1169]: time=2026-09-21T12:58:47.892Z level=INFO msg="Uploading 6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache (184B)"5468lb # [ 78.834173] niks3-server[1069]: 2026/09/21 12:58:47 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5469lb # [ 78.840989] niks3-server[1069]: 2026/09/21 12:58:47 INFO Registered completed upload object_key=nar/15y5akaw5jxrf5q28c25cfgfdz5zw27ihvpagi1mmvpnaw9na2a5.nar.zst5470lb # [ 78.844801] niks3-server[1069]: 2026/09/21 12:58:47 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.ls5471lb # [ 78.850061] niks3-server[1069]: 2026/09/21 12:58:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5472lb # [ 78.852366] niks3-server[1069]: 2026/09/21 12:58:47 INFO Signed narinfos id=26 count=25473lb # [ 78.854401] niks3-server[1069]: 2026/09/21 12:58:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5474lb # [ 78.856430] niks3-server[1069]: 2026/09/21 12:58:47 INFO Signed narinfos id=25 count=15475worker2 # [ 78.887923] nix-grpc-daemon[1169]: time=2026-09-21T12:58:47.950Z level=INFO msg="Uploading 3 narinfos"5476lb # [ 78.860352] niks3-server[1069]: 2026/09/21 12:58:47 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.ls5477lb # [ 78.870796] niks3-server[1069]: 2026/09/21 12:58:47 INFO Registered completed upload object_key=6zp5fijxpjrd2vv7855sb0z7aag6mm8d.narinfo5478lb # [ 78.877567] niks3-server[1069]: 2026/09/21 12:58:47 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5479lb # [ 78.880643] niks3-server[1069]: 2026/09/21 12:58:47 INFO Completed upload id=255480lb # [ 78.882275] niks3-server[1069]: 2026/09/21 12:58:47 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5481lb # [ 78.885428] niks3-server[1069]: 2026/09/21 12:58:47 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5482lb # [ 78.888367] niks3-server[1069]: 2026/09/21 12:58:47 INFO Completed upload id=265483worker2 # [ 78.917973] nix-grpc-daemon[1169]: time=2026-09-21T12:58:47.981Z level=INFO msg="Upload complete. (342ms)"5484lb # [ 78.889918] niks3-server[1069]: 2026/09/21 12:58:47 INFO Registered completed upload object_key=cmsrl1s0jpx66sskcbsv9jgl7ryk8101.narinfo5485worker2 # [ 78.922836] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56370 duration_s=0 paths=2 nar_bytes_in=3125486worker2 # [ 78.933547] nix-daemon[873]: reaped child process 1167, status = succeeded5487client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache >&2, in 1.30 seconds)5488worker2: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache5489worker2: (finished: must succeed: test -e /nix/store/cmsrl1s0jpx66sskcbsv9jgl7ryk8101-local-input-viacache && test -e /nix/store/6zp5fijxpjrd2vv7855sb0z7aag6mm8d-refers-to-input-viacache, in 0.03 seconds)5490worker2 # [ 79.012554] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:48Z level=info event=scheduler_connected addr=lb:500515491worker2 # [ 79.015661] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5492worker1 # [ 79.035806] systemd[1]: Listening on Nix worker-protocol over gRPC.5493worker1 # [ 79.042231] systemd[1]: Starting Nix worker-protocol over gRPC...5494worker2 # [ 79.216990] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:48Z level=info event=scheduler_connected addr=lb:500515495worker2 # [ 79.220898] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5496worker1 # [ 79.229951] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5497worker1 # [ 79.231795] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5498worker1 # [ 79.235858] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15499worker1 # [ 79.243062] nix-grpc-daemon[1227]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5500worker1 # [ 79.246108] nix-grpc-daemon[1227]: E0000 00:00:1789995528.086452 1227 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5501worker1 # [ 79.250519] nix-grpc-daemon[1227]: E0000 00:00:1789995528.091873 1227 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5502worker1 # [ 79.253650] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5503worker1 # [ 79.257239] systemd[1]: Started Nix worker-protocol over gRPC.5504lb: waiting until exactly no endpoints of x86_64-linux are unhealthy5505lb: must succeed: curl -sf localhost:9901/clusters5506lb # [ 79.257631] niks3-server[1069]: 2026/09/21 12:58:48 INFO lead: acquired remote=192.168.1.3:389745507worker1 # [ 79.265070] nix-grpc-daemon[1227]: E0000 00:00:1789995528.107172 1268 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5508worker1 # [ 79.268069] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=scheduler_take_over5509worker1 # [ 79.277171] nix-grpc-daemon[1227]: E0000 00:00:1789995528.120547 1268 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5510worker1 # [ 79.281155] nix-grpc-daemon[1227]: E0000 00:00:1789995528.122821 1268 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5511worker1 # [ 79.291241] nix-grpc-daemon[1227]: E0000 00:00:1789995528.134577 1268 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5512worker1 # [ 79.301097] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=scheduler_connected addr=lb:500515513worker1 # [ 79.302740] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5514lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5515worker2 # [ 79.422080] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:48Z level=info event=scheduler_connected addr=lb:500515516worker2 # [ 79.425895] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5517worker1 # [ 79.504245] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=scheduler_connected addr=lb:500515518worker1 # [ 79.508414] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5519worker2 # [ 79.626824] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:48Z level=info event=scheduler_connected addr=lb:500515520worker2 # [ 79.630655] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5521worker1 # [ 79.709788] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=scheduler_connected addr=lb:500515522worker1 # [ 79.713786] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5523worker2 # [ 79.831791] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:48Z level=info event=scheduler_connected addr=lb:500515524worker2 # [ 79.835637] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5525worker1 # [ 79.914752] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=scheduler_connected addr=lb:500515526worker1 # [ 79.922912] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:495805527worker1 # [ 79.926926] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05528worker2 # [ 80.037566] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:49Z level=info event=scheduler_connected addr=lb:500515529worker1 # [ 80.024841] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=oidc_jwks provider=mock keys=15530worker1 # [ 80.028913] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:495805531worker1 # [ 80.033317] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:48Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05532lb: must succeed: curl -sf localhost:9901/clusters5533lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5534lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.16 seconds)5535(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 8.32 seconds)5536subtest: a niks3 restart does not cost the leader its role5537lb: must succeed: curl -sf localhost:9901/clusters5538lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5539lb: must succeed: curl -sf localhost:9901/clusters5540lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5541lb # [ 80.562403] niks3-server[1069]: 2026/09/21 12:58:49 INFO Shutdown signal received, draining in-flight requests timeout=10s5542lb # [ 80.566274] systemd[1]: Stopping niks3 server...5543lb # [ 80.566985] niks3-server[1069]: 2026/09/21 12:58:49 INFO lead: released remote=192.168.1.3:389745544worker2 # [ 80.595514] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:49Z level=info event=scheduler_lock_lost status=2005545worker1 # [ 80.569975] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:49Z level=info event=scheduler_lock_lost status=2005546lb # [ 80.576804] systemd[1]: niks3.service: Deactivated successfully.5547lb # [ 80.578394] systemd[1]: Stopped niks3 server.5548lb # [ 80.579257] systemd[1]: niks3.service: Consumed 584ms CPU time over 49.536s wall clock time, 20.3M memory peak, 68.1K incoming IP traffic, 98.3K outgoing IP traffic.5549lb # [ 80.587750] systemd[1]: Starting niks3 server...5550lb # [ 80.655582] niks3-server[1174]: 2026/09/21 12:58:49 goose: no migrations to run. current version: 202609200000005551lb # [ 80.662152] niks3-server[1174]: 2026/09/21 12:58:49 OK 1_commit_pending_closure.sql (4.63ms)5552lb # [ 80.665781] niks3-server[1174]: 2026/09/21 12:58:49 OK 2_object_stats_trigger.sql (3.24ms)5553lb # [ 80.666973] niks3-server[1174]: 2026/09/21 12:58:49 goose: up to current file version: 25554lb # [ 80.668757] niks3-server[1174]: 2026/09/21 12:58:49 INFO Loaded signing key name=farm-test-1 path=/nix/store/c745bsy529madd2s6hz4z5is2436ni04-key5555lb # [ 80.680848] niks3-server[1174]: 2026/09/21 12:58:49 INFO Read proxy enabled — serving cache objects from S35556lb # [ 80.682420] niks3-server[1174]: 2026/09/21 12:58:49 INFO Using socket-activated listener address=0.0.0.0:57515557lb # [ 80.684723] niks3-server[1174]: 2026/09/21 12:58:49 INFO systemd watchdog enabled interval=15s5558lb # [ 80.686041] niks3-server[1174]: 2026/09/21 12:58:49 INFO Starting HTTP server address=0.0.0.0:57515559lb # [ 80.687238] systemd[1]: Started niks3 server.5560lb: waiting for unit niks3.service5561lb: (finished: waiting for unit niks3.service, in 0.04 seconds)5562lb: must succeed: curl -sf localhost:9901/clusters5563lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5564worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5565worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5566lb: must succeed: curl -sf localhost:9901/clusters5567lb # [ 80.871079] niks3-server[1174]: 2026/09/21 12:58:49 INFO lead: acquired remote=192.168.1.3:389885568lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5569lb: must succeed: curl -sf localhost:9901/clusters5570lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5571worker1: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5572worker1: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.06 seconds)5573client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&25574worker2 # [ 81.316638] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:50Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:563705575client # don't know how to build these paths:5576client # /nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5577worker1 # [ 81.303451] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495905578worker1 # [ 81.316416] nix-daemon[849]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)5579worker1 # [ 81.333908] nix-grpc-daemon[1227]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 34 ms (attempt 1/5)5580worker1 # [ 81.373698] nix-grpc-daemon[1227]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 85 ms (attempt 2/5)5581worker1 # [ 81.464655] nix-grpc-daemon[1227]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 361 ms (attempt 3/5)5582worker1 # [ 81.832569] nix-grpc-daemon[1227]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 169 ms (attempt 4/5)5583worker1 # [ 82.002356] nix-grpc-daemon[1227]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5584worker1 # [ 82.005346] nix-grpc-daemon[1227]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5585worker1 # [ 82.046348] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:50Z level=debug event=want client=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv cached=05586worker1 # [ 82.051377] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:50Z level=debug event=assigned drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv worker=192.168.1.3:50051 clients=15587worker1 # [ 82.096270] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495905588worker1 # [ 82.110560] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495905589client # copying 1 paths to 'lb:50051'...5590client # copying path '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' to 'lb:50051'...5591worker1 # [ 82.124168] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495905592worker1 # [ 82.134365] nix-daemon[849]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)5593worker1 # [ 82.156461] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:50Z level=info event=niks3_push_started pid=12995594worker1 # [ 82.376098] nix-grpc-daemon[1299]: time=2026-09-21T12:58:51.218Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""5595worker1 # [ 82.558095] nix-daemon[849]: accepted connection from pid 1312, user nix-grpc-daemon (trusted)5596worker1 # [ 82.570265] nix-daemon[849]: reaped child process 1315, status = succeeded5597worker1 # [ 82.752145] nix-daemon[849]: accepted connection from pid 1317, user nix-grpc-daemon (trusted)5598worker1 # [ 82.765669] nix-daemon[849]: reaped child process 1320, status = succeeded5599lb # [ 82.780666] niks3-server[1174]: 2026/09/21 12:58:51 INFO Received uploads request method=POST path=/api/pending_closures5600worker1 # [ 82.798698] nix-grpc-daemon[1299]: time=2026-09-21T12:58:51.642Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5601worker1 # [ 82.802131] nix-grpc-daemon[1299]: time=2026-09-21T12:58:51.643Z level=INFO msg="Uploading 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv (536B)"5602lb # [ 82.833848] niks3-server[1174]: 2026/09/21 12:58:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5603lb # [ 82.843691] niks3-server[1174]: 2026/09/21 12:58:51 INFO Registered completed upload object_key=nar/0wjrljh500qj2dh74gryz4ik45bn1iv9w72490z0f2w0l2x4c87h.nar.zst5604lb # [ 82.865772] niks3-server[1174]: 2026/09/21 12:58:51 INFO Signed narinfos id=27 count=15605worker1 # [ 82.871926] nix-grpc-daemon[1299]: time=2026-09-21T12:58:51.714Z level=INFO msg="Uploading 1 narinfos"5606lb # [ 82.876893] niks3-server[1174]: 2026/09/21 12:58:51 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.ls5607lb # [ 82.878844] niks3-server[1174]: 2026/09/21 12:58:51 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5608lb # [ 82.885139] niks3-server[1174]: 2026/09/21 12:58:51 INFO Registered completed upload object_key=6finahhb2q4vla8rjb6gi4z4cvf71qmc.narinfo5609lb # [ 82.888944] niks3-server[1174]: 2026/09/21 12:58:51 INFO Completed upload id=275610worker1 # [ 82.892584] nix-grpc-daemon[1299]: time=2026-09-21T12:58:51.735Z level=INFO msg="Upload complete. (512ms)"5611worker1 # [ 82.897767] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49590 duration_s=0 paths=1 nar_bytes_in=5365612worker1 # [ 82.904951] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495905613worker1 # [ 82.917116] nix-daemon[849]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)5614worker1 # [ 82.927317] nix-daemon[849]: reaped child process 1297, status = succeeded5615worker1 # [ 82.933601] nix-daemon[849]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)5616client # building '/nix/store/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv' on lb:50051...5617client # farm-a> node-a: building 6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5618client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5619client # farm-a> LOG-farm-a-after-niks3-restart5620worker1 # [ 84.348164] nix-daemon[849]: reaped child process 1324, status = succeeded5621worker1 # [ 84.512902] nix-daemon[849]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5622worker1 # [ 84.526622] nix-daemon[849]: reaped child process 1343, status = succeeded5623lb # [ 84.541693] niks3-server[1174]: 2026/09/21 12:58:53 INFO Received uploads request method=POST path=/api/pending_closures5624worker1 # [ 84.557393] nix-grpc-daemon[1299]: time=2026-09-21T12:58:53.400Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5625worker1 # [ 84.558939] nix-grpc-daemon[1299]: time=2026-09-21T12:58:53.402Z level=INFO msg="Uploading vllq1krpcvis69h349hy98vfz55ld1wv-farm-a (144B)"5626lb # [ 84.581751] niks3-server[1174]: 2026/09/21 12:58:53 INFO Registered completed upload object_key=log/6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv5627lb # [ 84.593482] niks3-server[1174]: 2026/09/21 12:58:53 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5628lb # [ 84.605747] niks3-server[1174]: 2026/09/21 12:58:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5629lb # [ 84.610104] niks3-server[1174]: 2026/09/21 12:58:53 INFO Signed narinfos id=28 count=15630worker1 # [ 84.614158] nix-grpc-daemon[1299]: time=2026-09-21T12:58:53.456Z level=INFO msg="Uploading 1 narinfos"5631lb # [ 84.615448] niks3-server[1174]: 2026/09/21 12:58:53 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.ls5632lb # [ 84.623891] niks3-server[1174]: 2026/09/21 12:58:53 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5633lb # [ 84.627748] niks3-server[1174]: 2026/09/21 12:58:53 INFO Completed upload id=285634worker1 # [ 84.631241] nix-grpc-daemon[1299]: time=2026-09-21T12:58:53.474Z level=INFO msg="Upload complete. (290ms)"5635lb # [ 84.630114] niks3-server[1174]: 2026/09/21 12:58:53 INFO Registered completed upload object_key=vllq1krpcvis69h349hy98vfz55ld1wv.narinfo5636worker1 # [ 84.641759] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49590 duration_s=1 drv=6finahhb2q4vla8rjb6gi4z4cvf71qmc-farm-a.drv assign_id=1 outputs=15637worker1 # [ 84.656275] nix-daemon[849]: reaped child process 1322, status = succeeded5638worker1 # [ 84.659117] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:53Z level=debug event=want client=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv cached=05639worker1 # [ 84.660965] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:53Z level=debug event=assigned drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv worker=192.168.1.3:50051 clients=15640worker1 # [ 84.706181] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495905641worker1 # [ 84.719819] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495905642worker1 # [ 84.734334] nix-grpc-daemon[1299]: time=2026-09-21T12:58:53.577Z level=INFO msg="All 1 paths already cached"5643client # copying 1 paths to 'lb:50051'...5644client # copying path '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' to 'lb:50051'...5645worker1 # [ 84.742609] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495905646worker1 # [ 84.753300] nix-daemon[849]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)5647worker1 # [ 84.943973] nix-daemon[849]: accepted connection from pid 1349, user nix-grpc-daemon (trusted)5648worker1 # [ 84.958060] nix-daemon[849]: reaped child process 1352, status = succeeded5649worker1 # [ 85.137923] nix-daemon[849]: accepted connection from pid 1354, user nix-grpc-daemon (trusted)5650worker1 # [ 85.152378] nix-daemon[849]: reaped child process 1357, status = succeeded5651lb # [ 85.165165] niks3-server[1174]: 2026/09/21 12:58:54 INFO Received uploads request method=POST path=/api/pending_closures5652worker1 # [ 85.178457] nix-grpc-daemon[1299]: time=2026-09-21T12:58:54.021Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5653worker1 # [ 85.180075] nix-grpc-daemon[1299]: time=2026-09-21T12:58:54.023Z level=INFO msg="Uploading 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv (656B)"5654lb # [ 85.195572] niks3-server[1174]: 2026/09/21 12:58:54 INFO Registered completed upload object_key=nar/0sjnwmbwn7i8g7g7czwn9r2pzaaa1j1yx3dfa4gw29ynhmpz9v3j.nar.zst5655lb # [ 85.205236] niks3-server[1174]: 2026/09/21 12:58:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5656lb # [ 85.209489] niks3-server[1174]: 2026/09/21 12:58:54 INFO Signed narinfos id=29 count=15657worker1 # [ 85.215662] nix-grpc-daemon[1299]: time=2026-09-21T12:58:54.059Z level=INFO msg="Uploading 1 narinfos"5658lb # [ 85.217606] niks3-server[1174]: 2026/09/21 12:58:54 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.ls5659lb # [ 85.229458] niks3-server[1174]: 2026/09/21 12:58:54 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5660lb # [ 85.237494] niks3-server[1174]: 2026/09/21 12:58:54 INFO Completed upload id=295661worker1 # [ 85.240851] nix-grpc-daemon[1299]: time=2026-09-21T12:58:54.084Z level=INFO msg="Upload complete. (475ms)"5662lb # [ 85.240586] niks3-server[1174]: 2026/09/21 12:58:54 INFO Registered completed upload object_key=5g2krgx4inyydhy0jrhhylv4xhhlm7j7.narinfo5663worker1 # [ 85.245307] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49590 duration_s=0 paths=1 nar_bytes_in=6565664worker1 # [ 85.249990] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495905665worker1 # [ 85.260248] nix-daemon[849]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)5666worker1 # [ 85.267136] nix-daemon[849]: reaped child process 1347, status = succeeded5667worker1 # [ 85.270973] nix-daemon[849]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)5668client # building '/nix/store/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv' on lb:50051...5669client # farm-b> node-a: building 5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5670client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5671client # farm-b> LOG-farm-b-after-niks3-restart5672worker1 # [ 86.098890] nix-daemon[849]: reaped child process 1361, status = succeeded5673worker1 # [ 86.285630] nix-daemon[849]: accepted connection from pid 1377, user nix-grpc-daemon (trusted)5674worker1 # [ 86.297796] nix-daemon[849]: reaped child process 1380, status = succeeded5675lb # [ 86.313120] niks3-server[1174]: 2026/09/21 12:58:55 INFO Received uploads request method=POST path=/api/pending_closures5676worker1 # [ 86.325039] nix-grpc-daemon[1299]: time=2026-09-21T12:58:55.168Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5677worker1 # [ 86.326608] nix-grpc-daemon[1299]: time=2026-09-21T12:58:55.170Z level=INFO msg="Uploading vkqg7lcax8h1mxs0g750ficqiii1falw-farm-b (192B)"5678lb # [ 86.354285] niks3-server[1174]: 2026/09/21 12:58:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5679lb # [ 86.361687] niks3-server[1174]: 2026/09/21 12:58:55 INFO Signed narinfos id=30 count=15680worker1 # [ 86.368106] nix-grpc-daemon[1299]: time=2026-09-21T12:58:55.211Z level=INFO msg="Uploading 1 narinfos"5681lb # [ 86.376153] niks3-server[1174]: 2026/09/21 12:58:55 INFO Registered completed upload object_key=nar/12zl6l8xhwx5hdivnvf1vw18yzvz3764mhdd9n03x81phzqblxj0.nar.zst5682lb # [ 86.381730] niks3-server[1174]: 2026/09/21 12:58:55 INFO Registered completed upload object_key=log/5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv5683lb # [ 86.386107] niks3-server[1174]: 2026/09/21 12:58:55 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.ls5684lb # [ 86.392851] niks3-server[1174]: 2026/09/21 12:58:55 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5685lb # [ 86.401701] niks3-server[1174]: 2026/09/21 12:58:55 INFO Completed upload id=305686lb # [ 86.403567] niks3-server[1174]: 2026/09/21 12:58:55 INFO Registered completed upload object_key=vkqg7lcax8h1mxs0g750ficqiii1falw.narinfo5687worker1 # [ 86.405319] nix-grpc-daemon[1299]: time=2026-09-21T12:58:55.248Z level=INFO msg="Upload complete. (327ms)"5688worker1 # [ 86.413861] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49590 duration_s=1 drv=5g2krgx4inyydhy0jrhhylv4xhhlm7j7-farm-b.drv assign_id=2 outputs=15689worker1 # [ 86.425132] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:55Z level=debug event=want client=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv cached=05690worker1 # [ 86.429213] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:55Z level=debug event=assigned drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv worker=192.168.1.3:50051 clients=15691worker1 # [ 86.438664] nix-daemon[849]: reaped child process 1359, status = succeeded5692worker1 # [ 86.468579] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495905693worker1 # [ 86.481726] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:495905694worker1 # [ 86.495708] nix-grpc-daemon[1299]: time=2026-09-21T12:58:55.339Z level=INFO msg="All 2 paths already cached"5695client # copying 1 paths to 'lb:50051'...5696client # copying path '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' to 'lb:50051'...5697worker1 # [ 86.504754] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:495905698worker1 # [ 86.513617] nix-daemon[849]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)5699worker1 # [ 86.702232] nix-daemon[849]: accepted connection from pid 1386, user nix-grpc-daemon (trusted)5700worker1 # [ 86.716367] nix-daemon[849]: reaped child process 1389, status = succeeded5701worker1 # [ 86.898192] nix-daemon[849]: accepted connection from pid 1391, user nix-grpc-daemon (trusted)5702worker1 # [ 86.912164] nix-daemon[849]: reaped child process 1394, status = succeeded5703lb # [ 86.924922] niks3-server[1174]: 2026/09/21 12:58:55 INFO Received uploads request method=POST path=/api/pending_closures5704worker1 # [ 86.937242] nix-grpc-daemon[1299]: time=2026-09-21T12:58:55.780Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5705worker1 # [ 86.938662] nix-grpc-daemon[1299]: time=2026-09-21T12:58:55.780Z level=INFO msg="Uploading v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv (784B)"5706lb # [ 86.952309] niks3-server[1174]: 2026/09/21 12:58:55 INFO Registered completed upload object_key=nar/0ppnnsmr641nka691g6vrp1crps1bbn31zm7nc5za9pl2y2iwb0f.nar.zst5707lb # [ 86.959975] niks3-server[1174]: 2026/09/21 12:58:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5708worker1 # [ 86.970720] nix-grpc-daemon[1299]: time=2026-09-21T12:58:55.813Z level=INFO msg="Uploading 1 narinfos"5709lb # [ 86.967143] niks3-server[1174]: 2026/09/21 12:58:55 INFO Signed narinfos id=31 count=15710lb # [ 86.971631] niks3-server[1174]: 2026/09/21 12:58:55 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.ls5711lb # [ 86.984969] niks3-server[1174]: 2026/09/21 12:58:55 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5712lb # [ 86.989460] niks3-server[1174]: 2026/09/21 12:58:55 INFO Completed upload id=315713worker1 # [ 86.995342] nix-grpc-daemon[1299]: time=2026-09-21T12:58:55.838Z level=INFO msg="Upload complete. (468ms)"5714lb # [ 86.992669] niks3-server[1174]: 2026/09/21 12:58:55 INFO Registered completed upload object_key=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c.narinfo5715worker1 # [ 86.998265] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49590 duration_s=0 paths=1 nar_bytes_in=7845716worker1 # [ 87.003296] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:495905717worker1 # [ 87.012298] nix-daemon[849]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)5718worker1 # [ 87.018416] nix-daemon[849]: reaped child process 1384, status = succeeded5719worker1 # [ 87.023841] nix-daemon[849]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)5720client # building '/nix/store/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv' on lb:50051...5721client # farm-top> node-a: building v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5722client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5723client # farm-top> LOG-farm-top-after-niks3-restart5724worker1 # [ 88.252219] nix-daemon[849]: reaped child process 1398, status = succeeded5725lb # [ 88.356075] envoy[922]: conn peer=[2001:db8:1::1]:36774 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1006 tx=305 ms=10215726worker1 # [ 88.438396] nix-daemon[849]: accepted connection from pid 1414, user nix-grpc-daemon (trusted)5727worker1 # [ 88.452484] nix-daemon[849]: reaped child process 1417, status = succeeded5728lb # [ 88.467462] niks3-server[1174]: 2026/09/21 12:58:57 INFO Received uploads request method=POST path=/api/pending_closures5729worker1 # [ 88.479237] nix-grpc-daemon[1299]: time=2026-09-21T12:58:57.322Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5730worker1 # [ 88.480759] nix-grpc-daemon[1299]: time=2026-09-21T12:58:57.324Z level=INFO msg="Uploading lyi0sw9pvmhznkzgfl39callq9c0n87r-farm-top (248B)"5731lb # [ 88.505399] niks3-server[1174]: 2026/09/21 12:58:57 INFO Registered completed upload object_key=log/v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv5732lb # [ 88.512516] niks3-server[1174]: 2026/09/21 12:58:57 INFO Registered completed upload object_key=nar/141220x0digv3f58fbh9c79avwg0fwgsk5jyac9ymkxmr8zs2prw.nar.zst5733lb # [ 88.524842] niks3-server[1174]: 2026/09/21 12:58:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5734worker1 # [ 88.534745] nix-grpc-daemon[1299]: time=2026-09-21T12:58:57.377Z level=INFO msg="Uploading 1 narinfos"5735lb # [ 88.531247] niks3-server[1174]: 2026/09/21 12:58:57 INFO Signed narinfos id=32 count=15736lb # [ 88.534531] niks3-server[1174]: 2026/09/21 12:58:57 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.ls5737lb # [ 88.544118] niks3-server[1174]: 2026/09/21 12:58:57 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5738lb # [ 88.550484] niks3-server[1174]: 2026/09/21 12:58:57 INFO Completed upload id=325739worker1 # [ 88.553981] nix-grpc-daemon[1299]: time=2026-09-21T12:58:57.397Z level=INFO msg="Upload complete. (321ms)"5740lb # [ 88.552393] niks3-server[1174]: 2026/09/21 12:58:57 INFO Registered completed upload object_key=lyi0sw9pvmhznkzgfl39callq9c0n87r.narinfo5741worker1 # [ 88.562225] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49590 duration_s=1 drv=v0ppd6w8hc0jlggdq33p3mhzyjib8r6c-farm-top.drv assign_id=3 outputs=15742worker1 # [ 88.575364] nix-daemon[849]: reaped child process 1396, status = succeeded5743client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart >&2, in 7.59 seconds)5744client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath5745client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag after-niks3-restart outPath, in 0.23 seconds)5746(finished: subtest: a niks3 restart does not cost the leader its role, in 8.42 seconds)5747subtest: scheduler down: the other node takes the lock and keeps it5748lb: must succeed: curl -sf localhost:9901/clusters5749lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5750lb: must succeed: curl -sf localhost:9901/clusters5751lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5752lb: must succeed: curl -sf localhost:9901/clusters5753lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5754lb: must succeed: curl -sf localhost:9901/clusters5755lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5756worker1 # [ 89.117614] systemd[1]: Stopping Nix worker-protocol over gRPC...5757worker1 # [ 89.256107] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:58Z level=info event=signal_exit5758lb # [ 90.083915] niks3-server[1174]: 2026/09/21 12:58:58 INFO lead: released remote=192.168.1.3:389885759worker2 # [ 90.415649] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:59Z level=info event=scheduler_restarting addr=lb:500515760worker1 # [ 90.393791] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:59Z level=info event=scheduler_restarting5761worker1 # [ 90.397601] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:59Z level=info event=scheduler_restarting addr=lb:500515762worker1 # [ 90.400828] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:59Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:495805763worker1 # [ 90.404423] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:59Z level=info event=worker_gone addr=192.168.1.4:500515764worker1 # [ 90.407714] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:59Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:495805765worker1 # [ 90.411740] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:59Z level=info event=worker_gone addr=192.168.1.3:500515766worker1 # [ 90.427248] nix-daemon[849]: reaped child process 1289, status = succeeded5767worker2 # [ 90.615810] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:59Z level=info event=scheduler_connected addr=lb:500515768worker2 # [ 90.619850] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5769worker1 # [ 90.604564] nix-grpc-daemon[1227]: ts=2026-09-21T12:58:59Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5770worker2 # [ 90.821141] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:59Z level=info event=scheduler_connected addr=lb:500515771worker2 # [ 90.825097] nix-grpc-daemon[1109]: ts=2026-09-21T12:58:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5772worker1 # [ 90.897652] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5773worker1 # [ 90.900952] systemd[1]: Stopped Nix worker-protocol over gRPC.5774worker1 # [ 90.905308] systemd[1]: nix-grpc-daemon.service: Consumed 1.106s CPU time over 11.854s wall clock time, 35.2M memory peak, 36.6K incoming IP traffic, 59.7K outgoing IP traffic.5775worker1 # [ 90.915964] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5776worker1 # [ 90.919331] systemd[1]: Closed Nix worker-protocol over gRPC.5777worker1 # [ 90.922142] systemd[1]: nix-grpc-daemon.socket: Consumed 564us CPU time over 11.891s wall clock time, 256K memory peak, 34.7K incoming IP traffic, 25.5K outgoing IP traffic.5778worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5779worker2 # [ 91.025629] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:00Z level=info event=scheduler_connected addr=lb:500515780worker2 # [ 91.027281] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5781worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds)5782worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5783worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5784worker2 # [ 91.228532] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:00Z level=info event=scheduler_connected addr=lb:500515785worker2 # [ 91.232460] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5786worker2 # [ 91.434327] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:00Z level=info event=scheduler_connected addr=lb:500515787worker2 # [ 91.438090] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5788worker2 # [ 91.639354] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:00Z level=info event=scheduler_connected addr=lb:500515789worker2 # [ 91.643268] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5790worker2 # [ 91.844290] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:00Z level=info event=scheduler_connected addr=lb:500515791worker2 # [ 91.848259] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5792worker2 # [ 92.049329] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:01Z level=info event=scheduler_connected addr=lb:500515793worker2 # [ 92.053261] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5794worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5795worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5796worker2 # [ 92.254088] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:01Z level=info event=scheduler_connected addr=lb:500515797worker2 # [ 92.257950] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5798worker2 # [ 92.459677] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:01Z level=info event=scheduler_connected addr=lb:500515799worker2 # [ 92.463463] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5800worker2 # [ 92.664592] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:01Z level=info event=scheduler_connected addr=lb:500515801worker2 # [ 92.668472] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5802worker2 # [ 92.869673] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:01Z level=info event=scheduler_connected addr=lb:500515803worker2 # [ 92.873568] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5804worker2 # [ 93.074601] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:02Z level=info event=scheduler_connected addr=lb:500515805worker2 # [ 93.078480] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5806worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5807worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5808worker2 # [ 93.279570] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:02Z level=info event=scheduler_connected addr=lb:500515809worker2 # [ 93.283453] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5810worker2 # [ 93.485386] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:02Z level=info event=scheduler_connected addr=lb:500515811worker2 # [ 93.488855] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5812lb # [ 93.573571] niks3-server[1174]: 2026/09/21 12:59:02 INFO lead: acquired remote=192.168.1.4:542105813worker2 # [ 93.606199] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:02Z level=info event=scheduler_take_over5814worker2 # [ 93.689874] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:02Z level=info event=scheduler_connected addr=lb:500515815worker2 # [ 93.693785] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5816worker2 # [ 93.894255] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:02Z level=info event=scheduler_connected addr=lb:500515817worker2 # [ 93.895951] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5818worker2 # [ 94.097234] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:03Z level=info event=scheduler_connected addr=lb:500515819worker2 # [ 94.114137] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:03Z level=info event=oidc_jwks provider=mock keys=15820worker2 # [ 94.118375] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:03Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:366965821worker2 # [ 94.122274] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:03Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=05822worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5823worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5824client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&25825client # don't know how to build these paths:5826client # /nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5827worker2 # [ 94.533858] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:563705828worker2 # [ 94.539526] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563705829worker2 # [ 94.561281] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:03Z level=debug event=want client=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv cached=05830worker2 # [ 94.565950] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:03Z level=debug event=assigned drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv worker=192.168.1.4:50051 clients=15831worker2 # [ 94.610349] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563705832worker2 # [ 94.622371] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563705833client # copying 1 paths to 'lb:50051'...5834client # copying path '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' to 'lb:50051'...5835worker2 # [ 94.634466] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563705836worker2 # [ 94.645141] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)5837worker2 # [ 94.839828] nix-daemon[873]: accepted connection from pid 1214, user nix-grpc-daemon (trusted)5838worker2 # [ 94.851577] nix-daemon[873]: reaped child process 1217, status = succeeded5839worker2 # [ 95.042599] nix-daemon[873]: accepted connection from pid 1219, user nix-grpc-daemon (trusted)5840worker2 # [ 95.054111] nix-daemon[873]: reaped child process 1222, status = succeeded5841lb # [ 95.036776] niks3-server[1174]: 2026/09/21 12:59:03 INFO Received uploads request method=POST path=/api/pending_closures5842worker2 # [ 95.075654] nix-grpc-daemon[1169]: time=2026-09-21T12:59:04.138Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5843worker2 # [ 95.076990] nix-grpc-daemon[1169]: time=2026-09-21T12:59:04.138Z level=INFO msg="Uploading 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv (520B)"5844lb # [ 95.062954] niks3-server[1174]: 2026/09/21 12:59:03 INFO Registered completed upload object_key=nar/013ynk63lqgbvdpr2qqb6pflhzfcrl005f3zkvnag3x7hqxjhibs.nar.zst5845lb # [ 95.071910] niks3-server[1174]: 2026/09/21 12:59:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5846worker2 # [ 95.109837] nix-grpc-daemon[1169]: time=2026-09-21T12:59:04.173Z level=INFO msg="Uploading 1 narinfos"5847lb # [ 95.080193] niks3-server[1174]: 2026/09/21 12:59:03 INFO Signed narinfos id=33 count=15848lb # [ 95.093233] niks3-server[1174]: 2026/09/21 12:59:04 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.ls5849lb # [ 95.096759] niks3-server[1174]: 2026/09/21 12:59:04 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5850lb # [ 95.104217] niks3-server[1174]: 2026/09/21 12:59:04 INFO Completed upload id=335851worker2 # [ 95.136175] nix-grpc-daemon[1169]: time=2026-09-21T12:59:04.199Z level=INFO msg="Upload complete. (479ms)"5852lb # [ 95.107484] niks3-server[1174]: 2026/09/21 12:59:04 INFO Registered completed upload object_key=0i80y1x9s0d06s86lhapa650pgsxgi0g.narinfo5853worker2 # [ 95.140325] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56370 duration_s=0 paths=1 nar_bytes_in=5205854worker2 # [ 95.144847] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563705855worker2 # [ 95.154261] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)5856worker2 # [ 95.156989] nix-daemon[873]: reaped child process 1212, status = succeeded5857worker2 # [ 95.162117] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)5858client # building '/nix/store/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv' on lb:50051...5859client # farm-a> worker2: building 0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5860client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5861client # farm-a> LOG-farm-a-on-standby5862worker2 # [ 95.989822] nix-daemon[873]: reaped child process 1226, status = succeeded5863worker2 # [ 96.142809] nix-daemon[873]: accepted connection from pid 1242, user nix-grpc-daemon (trusted)5864worker2 # [ 96.154633] nix-daemon[873]: reaped child process 1245, status = succeeded5865lb # [ 96.142548] niks3-server[1174]: 2026/09/21 12:59:05 INFO Received uploads request method=POST path=/api/pending_closures5866worker2 # [ 96.180312] nix-grpc-daemon[1169]: time=2026-09-21T12:59:05.243Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5867worker2 # [ 96.181965] nix-grpc-daemon[1169]: time=2026-09-21T12:59:05.245Z level=INFO msg="Uploading fp54697jryq5rvfg8ml51wzkr9xjvhrg-farm-a (136B)"5868lb # [ 96.177580] niks3-server[1174]: 2026/09/21 12:59:05 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5869lb # [ 96.186630] niks3-server[1174]: 2026/09/21 12:59:05 INFO Registered completed upload object_key=log/0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv5870worker2 # [ 96.226149] nix-grpc-daemon[1169]: time=2026-09-21T12:59:05.289Z level=INFO msg="Uploading 1 narinfos"5871lb # [ 96.197154] niks3-server[1174]: 2026/09/21 12:59:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5872lb # [ 96.200729] niks3-server[1174]: 2026/09/21 12:59:05 INFO Signed narinfos id=34 count=15873lb # [ 96.207270] niks3-server[1174]: 2026/09/21 12:59:05 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.ls5874lb # [ 96.216260] niks3-server[1174]: 2026/09/21 12:59:05 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5875lb # [ 96.224108] niks3-server[1174]: 2026/09/21 12:59:05 INFO Completed upload id=345876worker2 # [ 96.253642] nix-grpc-daemon[1169]: time=2026-09-21T12:59:05.317Z level=INFO msg="Upload complete. (284ms)"5877lb # [ 96.226772] niks3-server[1174]: 2026/09/21 12:59:05 INFO Registered completed upload object_key=fp54697jryq5rvfg8ml51wzkr9xjvhrg.narinfo5878worker2 # [ 96.264235] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56370 duration_s=1 drv=0i80y1x9s0d06s86lhapa650pgsxgi0g-farm-a.drv assign_id=1 outputs=15879worker2 # [ 96.273778] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:05Z level=debug event=want client=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv cached=05880worker2 # [ 96.277458] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:05Z level=debug event=assigned drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv worker=192.168.1.4:50051 clients=15881worker2 # [ 96.286151] nix-daemon[873]: reaped child process 1224, status = succeeded5882worker2 # [ 96.318484] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563705883worker2 # [ 96.329870] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563705884worker2 # [ 96.408518] nix-grpc-daemon[1169]: time=2026-09-21T12:59:05.471Z level=INFO msg="All 1 paths already cached"5885client # copying 1 paths to 'lb:50051'...5886client # copying path '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' to 'lb:50051'...5887worker2 # [ 96.417226] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563705888worker2 # [ 96.427165] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)5889worker2 # [ 96.632650] nix-daemon[873]: accepted connection from pid 1251, user nix-grpc-daemon (trusted)5890worker2 # [ 96.644306] nix-daemon[873]: reaped child process 1254, status = succeeded5891worker2 # [ 96.831510] nix-daemon[873]: accepted connection from pid 1256, user nix-grpc-daemon (trusted)5892worker2 # [ 96.842905] nix-daemon[873]: reaped child process 1259, status = succeeded5893lb # [ 96.831380] niks3-server[1174]: 2026/09/21 12:59:05 INFO Received uploads request method=POST path=/api/pending_closures5894worker2 # [ 96.869890] nix-grpc-daemon[1169]: time=2026-09-21T12:59:05.933Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5895worker2 # [ 96.871475] nix-grpc-daemon[1169]: time=2026-09-21T12:59:05.934Z level=INFO msg="Uploading 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv (640B)"5896lb # [ 96.861529] niks3-server[1174]: 2026/09/21 12:59:05 INFO Registered completed upload object_key=nar/12kaxaz59hi3nqzgsxphwdpbajdwrfjqknsv96cnrhpibnnzjlg0.nar.zst5897lb # [ 96.871235] niks3-server[1174]: 2026/09/21 12:59:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5898lb # [ 96.876149] niks3-server[1174]: 2026/09/21 12:59:05 INFO Signed narinfos id=35 count=15899worker2 # [ 96.908540] nix-grpc-daemon[1169]: time=2026-09-21T12:59:05.971Z level=INFO msg="Uploading 1 narinfos"5900lb # [ 96.882947] niks3-server[1174]: 2026/09/21 12:59:05 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.ls5901lb # [ 96.892466] niks3-server[1174]: 2026/09/21 12:59:05 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5902lb # [ 96.899542] niks3-server[1174]: 2026/09/21 12:59:05 INFO Completed upload id=355903worker2 # [ 96.929173] nix-grpc-daemon[1169]: time=2026-09-21T12:59:05.992Z level=INFO msg="Upload complete. (489ms)"5904lb # [ 96.902160] niks3-server[1174]: 2026/09/21 12:59:05 INFO Registered completed upload object_key=6532q0zszdl4cnys1gb39vpngyzd7y8j.narinfo5905worker2 # [ 96.933184] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56370 duration_s=0 paths=1 nar_bytes_in=6405906worker2 # [ 96.938204] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563705907worker2 # [ 96.947121] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)5908worker2 # [ 96.951877] nix-daemon[873]: reaped child process 1249, status = succeeded5909worker2 # [ 96.956797] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)5910client # building '/nix/store/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv' on lb:50051...5911client # farm-b> worker2: building 6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5912client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5913client # farm-b> LOG-farm-b-on-standby5914worker2 # [ 97.779973] nix-daemon[873]: reaped child process 1263, status = succeeded5915worker2 # [ 97.932087] nix-daemon[873]: accepted connection from pid 1279, user nix-grpc-daemon (trusted)5916worker2 # [ 97.943048] nix-daemon[873]: reaped child process 1282, status = succeeded5917lb # [ 97.923732] niks3-server[1174]: 2026/09/21 12:59:06 INFO Received uploads request method=POST path=/api/pending_closures5918worker2 # [ 97.961407] nix-grpc-daemon[1169]: time=2026-09-21T12:59:07.024Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5919worker2 # [ 97.963392] nix-grpc-daemon[1169]: time=2026-09-21T12:59:07.026Z level=INFO msg="Uploading 593ccq0xcckfsdii9xb4x5h0bapmahkp-farm-b (184B)"5920lb # [ 97.959223] niks3-server[1174]: 2026/09/21 12:59:06 INFO Registered completed upload object_key=nar/0k21bb46wfcxlq6p91pdvlzw7kkc20p87xpxr7zq5c5zr17k6rda.nar.zst5921lb # [ 97.969850] niks3-server[1174]: 2026/09/21 12:59:06 INFO Registered completed upload object_key=log/6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv5922lb # [ 97.975863] niks3-server[1174]: 2026/09/21 12:59:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5923lb # [ 97.981062] niks3-server[1174]: 2026/09/21 12:59:06 INFO Signed narinfos id=36 count=15924worker2 # [ 98.013356] nix-grpc-daemon[1169]: time=2026-09-21T12:59:07.076Z level=INFO msg="Uploading 1 narinfos"5925lb # [ 97.987530] niks3-server[1174]: 2026/09/21 12:59:06 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.ls5926lb # [ 97.996522] niks3-server[1174]: 2026/09/21 12:59:06 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5927lb # [ 98.003207] niks3-server[1174]: 2026/09/21 12:59:06 INFO Completed upload id=365928worker2 # [ 98.032585] nix-grpc-daemon[1169]: time=2026-09-21T12:59:07.095Z level=INFO msg="Upload complete. (267ms)"5929lb # [ 98.006105] niks3-server[1174]: 2026/09/21 12:59:06 INFO Registered completed upload object_key=593ccq0xcckfsdii9xb4x5h0bapmahkp.narinfo5930worker2 # [ 98.039335] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56370 duration_s=1 drv=6532q0zszdl4cnys1gb39vpngyzd7y8j-farm-b.drv assign_id=2 outputs=15931worker2 # [ 98.049842] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:07Z level=debug event=want client=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv cached=05932worker2 # [ 98.054229] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:07Z level=debug event=assigned drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv worker=192.168.1.4:50051 clients=15933worker2 # [ 98.062149] nix-daemon[873]: reaped child process 1261, status = succeeded5934worker2 # [ 98.093345] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563705935worker2 # [ 98.105508] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563705936worker2 # [ 98.118095] nix-grpc-daemon[1169]: time=2026-09-21T12:59:07.181Z level=INFO msg="All 2 paths already cached"5937client # copying 1 paths to 'lb:50051'...5938client # copying path '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' to 'lb:50051'...5939worker2 # [ 98.126278] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563705940worker2 # [ 98.134218] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)5941worker2 # [ 98.274112] nix-daemon[873]: accepted connection from pid 1288, user nix-grpc-daemon (trusted)5942worker2 # [ 98.285768] nix-daemon[873]: reaped child process 1291, status = succeeded5943lb # [ 98.356525] envoy[922]: conn peer=[2001:db8:1::1]:36786 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9726 tx=4120 ms=72815944lb # [ 98.360473] envoy[922]: conn peer=[2001:db8:1::3]:57882 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1726 tx=592 ms=112965945worker2 # [ 98.415954] nix-daemon[873]: accepted connection from pid 1293, user nix-grpc-daemon (trusted)5946worker2 # [ 98.427434] nix-daemon[873]: reaped child process 1296, status = succeeded5947lb # [ 98.408932] niks3-server[1174]: 2026/09/21 12:59:07 INFO Received uploads request method=POST path=/api/pending_closures5948worker2 # [ 98.447442] nix-grpc-daemon[1169]: time=2026-09-21T12:59:07.510Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5949worker2 # [ 98.448817] nix-grpc-daemon[1169]: time=2026-09-21T12:59:07.510Z level=INFO msg="Uploading bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv (768B)"5950lb # [ 98.434227] niks3-server[1174]: 2026/09/21 12:59:07 INFO Registered completed upload object_key=nar/0y5h6gmqp28k8bkzgrmc35pj8c8qhdi5bc0l17s14j217c1rzscm.nar.zst5951lb # [ 98.447392] niks3-server[1174]: 2026/09/21 12:59:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5952lb # [ 98.453553] niks3-server[1174]: 2026/09/21 12:59:07 INFO Signed narinfos id=37 count=15953worker2 # [ 98.485185] nix-grpc-daemon[1169]: time=2026-09-21T12:59:07.548Z level=INFO msg="Uploading 1 narinfos"5954lb # [ 98.456963] niks3-server[1174]: 2026/09/21 12:59:07 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.ls5955lb # [ 98.466590] niks3-server[1174]: 2026/09/21 12:59:07 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5956lb # [ 98.470589] niks3-server[1174]: 2026/09/21 12:59:07 INFO Completed upload id=375957worker2 # [ 98.501174] nix-grpc-daemon[1169]: time=2026-09-21T12:59:07.564Z level=INFO msg="Upload complete. (354ms)"5958lb # [ 98.472813] niks3-server[1174]: 2026/09/21 12:59:07 INFO Registered completed upload object_key=bgavyqwkalx49wvzslr2znbhxfj7h1mb.narinfo5959worker2 # [ 98.503032] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56370 duration_s=0 paths=1 nar_bytes_in=7685960worker2 # [ 98.506532] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563705961worker2 # [ 98.512277] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)5962worker2 # [ 98.515154] nix-daemon[873]: reaped child process 1286, status = succeeded5963client # building '/nix/store/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv' on lb:50051...5964client # farm-top> worker2: building bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5965worker2 # [ 98.520078] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)5966client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5967client # farm-top> LOG-farm-top-on-standby5968worker2 # [ 99.946265] nix-daemon[873]: reaped child process 1300, status = succeeded5969worker2 # [ 100.111569] nix-daemon[873]: accepted connection from pid 1316, user nix-grpc-daemon (trusted)5970worker2 # [ 100.123224] nix-daemon[873]: reaped child process 1319, status = succeeded5971lb # [ 100.109784] niks3-server[1174]: 2026/09/21 12:59:09 INFO Received uploads request method=POST path=/api/pending_closures5972worker2 # [ 100.146904] nix-grpc-daemon[1169]: time=2026-09-21T12:59:09.210Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5973worker2 # [ 100.148355] nix-grpc-daemon[1169]: time=2026-09-21T12:59:09.211Z level=INFO msg="Uploading bna0adh7g5jsp1sgpmaqnl7brapllfgc-farm-top (240B)"5974lb # [ 100.144813] niks3-server[1174]: 2026/09/21 12:59:09 INFO Registered completed upload object_key=nar/0bglib3l3asq436b8p5hj4yyk8c77b2qg09pcdiw1w2dfpvnd026.nar.zst5975lb # [ 100.156895] niks3-server[1174]: 2026/09/21 12:59:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5976lb # [ 100.161992] niks3-server[1174]: 2026/09/21 12:59:09 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.ls5977worker2 # [ 100.197723] nix-grpc-daemon[1169]: time=2026-09-21T12:59:09.261Z level=INFO msg="Uploading 1 narinfos"5978lb # [ 100.168124] niks3-server[1174]: 2026/09/21 12:59:09 INFO Signed narinfos id=38 count=15979lb # [ 100.174872] niks3-server[1174]: 2026/09/21 12:59:09 INFO Registered completed upload object_key=log/bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv5980lb # [ 100.184343] niks3-server[1174]: 2026/09/21 12:59:09 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5981lb # [ 100.190463] niks3-server[1174]: 2026/09/21 12:59:09 INFO Completed upload id=385982worker2 # [ 100.220189] nix-grpc-daemon[1169]: time=2026-09-21T12:59:09.283Z level=INFO msg="Upload complete. (292ms)"5983lb # [ 100.192399] niks3-server[1174]: 2026/09/21 12:59:09 INFO Registered completed upload object_key=bna0adh7g5jsp1sgpmaqnl7brapllfgc.narinfo5984worker2 # [ 100.226069] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56370 duration_s=1 drv=bgavyqwkalx49wvzslr2znbhxfj7h1mb-farm-top.drv assign_id=3 outputs=15985worker2 # [ 100.237998] nix-daemon[873]: reaped child process 1298, status = succeeded5986client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby >&2, in 5.96 seconds)5987client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath5988client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag on-standby outPath, in 0.17 seconds)5989worker1 # [ 100.475594] systemd[1]: Listening on Nix worker-protocol over gRPC.5990worker1 # [ 100.483719] systemd[1]: Starting Nix worker-protocol over gRPC...5991worker1 # [ 100.663626] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:09Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5992worker1 # [ 100.665538] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:09Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5993worker1 # [ 100.669311] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:09Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15994worker1 # [ 100.677065] nix-grpc-daemon[1435]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5995worker1 # [ 100.679178] nix-grpc-daemon[1435]: E0000 00:00:1789995549.519789 1435 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5996worker1 # [ 100.684042] nix-grpc-daemon[1435]: E0000 00:00:1789995549.524984 1435 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5997worker1 # [ 100.687991] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:09Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5998worker1 # [ 100.690922] systemd[1]: Started Nix worker-protocol over gRPC.5999lb: must succeed: curl -sf localhost:9901/clusters6000worker1 # [ 100.697136] nix-grpc-daemon[1435]: E0000 00:00:1789995549.540569 1475 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6001worker1 # [ 100.709327] nix-grpc-daemon[1435]: E0000 00:00:1789995549.552683 1475 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6002worker1 # [ 100.713129] nix-grpc-daemon[1435]: E0000 00:00:1789995549.554878 1475 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6003worker1 # [ 100.723443] nix-grpc-daemon[1435]: E0000 00:00:1789995549.566818 1475 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6004worker1 # [ 100.734476] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:09Z level=info event=scheduler_connected addr=lb:500516005worker2 # [ 100.762386] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:09Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:366966006worker2 # [ 100.766233] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:09Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06007lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6008worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6009worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6010lb: must succeed: curl -sf localhost:9901/clusters6011lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6012lb: must succeed: curl -sf localhost:9901/clusters6013lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6014worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6015worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6016(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.16 seconds)6017subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise6018client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs6019client # [ 101.068474] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs.6020client # Running as unit: rs.service; invocation ID: df4080453e694b10a657e36e8054920b6021client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag rs, in 0.07 seconds)6022worker1 # [ 101.286621] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:459706023worker1 # [ 101.295933] nix-daemon[849]: accepted connection from pid 1435, user nix-grpc-daemon (trusted)6024client # [ 101.324667] nix[1202]: don't know how to build these paths:6025client # [ 101.325561] nix[1202]: /nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6026worker2 # [ 101.345884] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:10Z level=debug event=want client=2 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv cached=06027worker2 # [ 101.349786] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:10Z level=debug event=assigned drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv worker=192.168.1.4:50051 clients=16028worker2 # [ 101.388434] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563706029worker2 # [ 101.400545] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563706030client # [ 101.398213] nix[1202]: copying 1 paths to 'lb:50051'...6031worker2 # [ 101.411665] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563706032client # [ 101.400350] nix[1202]: copying path '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' to 'lb:50051'...6033worker2 # [ 101.423978] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)6034worker2 # [ 101.598943] nix-daemon[873]: accepted connection from pid 1332, user nix-grpc-daemon (trusted)6035worker2 # [ 101.610716] nix-daemon[873]: reaped child process 1335, status = succeeded6036worker2 # [ 101.767428] nix-daemon[873]: accepted connection from pid 1337, user nix-grpc-daemon (trusted)6037worker2 # [ 101.778903] nix-daemon[873]: reaped child process 1340, status = succeeded6038lb # [ 101.761288] niks3-server[1174]: 2026/09/21 12:59:10 INFO Received uploads request method=POST path=/api/pending_closures6039worker2 # [ 101.801063] nix-grpc-daemon[1169]: time=2026-09-21T12:59:10.864Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6040worker2 # [ 101.802473] nix-grpc-daemon[1169]: time=2026-09-21T12:59:10.864Z level=INFO msg="Uploading 2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv (416B)"6041lb # [ 101.787537] niks3-server[1174]: 2026/09/21 12:59:10 INFO Registered completed upload object_key=nar/1p73bzh7zqz68fm48xgidr9p1hfd08x3vp8znnaaycsmagbr8hl6.nar.zst6042lb # [ 101.799598] niks3-server[1174]: 2026/09/21 12:59:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign6043lb # [ 101.804737] niks3-server[1174]: 2026/09/21 12:59:10 INFO Signed narinfos id=39 count=16044worker2 # [ 101.836855] nix-grpc-daemon[1169]: time=2026-09-21T12:59:10.900Z level=INFO msg="Uploading 1 narinfos"6045lb # [ 101.811731] niks3-server[1174]: 2026/09/21 12:59:10 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.ls6046lb # [ 101.822905] niks3-server[1174]: 2026/09/21 12:59:10 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete6047lb # [ 101.829476] niks3-server[1174]: 2026/09/21 12:59:10 INFO Completed upload id=396048worker2 # [ 101.859123] nix-grpc-daemon[1169]: time=2026-09-21T12:59:10.922Z level=INFO msg="Upload complete. (423ms)"6049lb # [ 101.832499] niks3-server[1174]: 2026/09/21 12:59:10 INFO Registered completed upload object_key=2765168rrfa4dbs7pfx66j88710jx2fm.narinfo6050worker2 # [ 101.863147] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56370 duration_s=0 paths=1 nar_bytes_in=4166051worker2 # [ 101.868208] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563706052worker2 # [ 101.880155] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)6053worker2 # [ 101.882126] nix-daemon[873]: reaped child process 1330, status = succeeded6054client # [ 101.873652] nix[1202]: building '/nix/store/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv' on lb:50051...6055worker2 # [ 101.886807] nix-daemon[873]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)6056lb: must succeed: curl -sf localhost:9901/clusters6057lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6058lb: must succeed: curl -sf localhost:9901/clusters6059lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6060lb: must succeed: curl -sf localhost:9901/clusters6061lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6062lb: must succeed: curl -sf localhost:9901/clusters6063lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6064worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true6065worker1 # [ 103.975161] systemd-journald[287]: Received client request to rotate journal, rotating.6066worker1 # [ 103.993595] systemd-journald[287]: Vacuuming done, freed 0B of archived journals from /var/log/journal/ca8526a882954ca5a423ea0baee0d16a.6067worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.07 seconds)6068worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service6069worker2 # [ 103.930651] systemd[1]: Stopping Nix worker-protocol over gRPC...6070worker2 # [ 103.932460] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:12Z level=info event=draining6071worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)6072client: must succeed: sleep 5.06073lb # [ 108.356907] envoy[922]: conn peer=[2001:db8:1::1]:35552 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9590 tx=4079 ms=57046074client: (finished: must succeed: sleep 5.0, in 5.04 seconds)6075worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='6076worker2 # [ 109.307451] nix-daemon[873]: reaped child process 1344, status = succeeded6077worker2 # [ 109.456675] nix-daemon[873]: accepted connection from pid 1385, user nix-grpc-daemon (trusted)6078worker2 # [ 109.467255] nix-daemon[873]: reaped child process 1388, status = succeeded6079lb # [ 109.453685] niks3-server[1174]: 2026/09/21 12:59:18 INFO Received uploads request method=POST path=/api/pending_closures6080worker2 # [ 109.493234] nix-grpc-daemon[1169]: time=2026-09-21T12:59:18.556Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6081worker2 # [ 109.494817] nix-grpc-daemon[1169]: time=2026-09-21T12:59:18.558Z level=INFO msg="Uploading 6qby8qgv9vwglq2yryw4zqyixrizh5d2-slow-rs (120B)"6082lb # [ 109.491951] niks3-server[1174]: 2026/09/21 12:59:18 INFO Registered completed upload object_key=log/2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv6083lb # [ 109.500156] niks3-server[1174]: 2026/09/21 12:59:18 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst6084lb # [ 109.511640] niks3-server[1174]: 2026/09/21 12:59:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign6085worker2 # [ 109.546514] nix-grpc-daemon[1169]: time=2026-09-21T12:59:18.609Z level=INFO msg="Uploading 1 narinfos"6086lb # [ 109.517174] niks3-server[1174]: 2026/09/21 12:59:18 INFO Signed narinfos id=40 count=16087lb # [ 109.523261] niks3-server[1174]: 2026/09/21 12:59:18 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.ls6088lb # [ 109.532848] niks3-server[1174]: 2026/09/21 12:59:18 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete6089lb # [ 109.538824] niks3-server[1174]: 2026/09/21 12:59:18 INFO Completed upload id=406090lb # [ 109.541443] niks3-server[1174]: 2026/09/21 12:59:18 INFO Registered completed upload object_key=6qby8qgv9vwglq2yryw4zqyixrizh5d2.narinfo6091worker2 # [ 109.569999] nix-grpc-daemon[1169]: time=2026-09-21T12:59:18.633Z level=INFO msg="Upload complete. (269ms)"6092worker2 # [ 109.577899] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56370 duration_s=7 drv=2765168rrfa4dbs7pfx66j88710jx2fm-slow-rs.drv assign_id=4 outputs=16093worker2 # [ 109.591159] nix-daemon[873]: reaped child process 1342, status = succeeded6094client # [ 109.607426] systemd[1]: rs.service: Deactivated successfully.6095client # [ 109.611823] systemd[1]: rs.service: Consumed 126ms CPU time over 8.541s wall clock time, 41.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.6096worker2 # [ 109.932500] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:18Z level=info event=signal_exit6097lb # [ 110.803862] niks3-server[1174]: 2026/09/21 12:59:19 INFO lead: released remote=192.168.1.4:542106098worker1 # [ 111.109736] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:19Z level=info event=scheduler_restarting addr=lb:500516099worker2 # [ 111.136867] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:20Z level=info event=scheduler_restarting6100worker2 # [ 111.141847] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:20Z level=info event=scheduler_restarting addr=lb:500516101worker2 # [ 111.144844] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:20Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:366966102worker2 # [ 111.148828] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:20Z level=info event=worker_gone addr=192.168.1.3:500516103worker2 # [ 111.151920] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:20Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:366966104worker2 # [ 111.155339] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:20Z level=info event=worker_gone addr=192.168.1.4:500516105worker2 # [ 111.173490] nix-daemon[873]: reaped child process 1161, status = succeeded6106worker1 # [ 111.310553] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:20Z level=info event=scheduler_connected addr=lb:500516107worker1 # [ 111.314676] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6108worker2 # [ 111.348196] nix-grpc-daemon[1109]: ts=2026-09-21T12:59:20Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6109worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.20 seconds)6110client: waiting for success: ! systemctl is-active rs6111client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)6112client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }6113client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.04 seconds)6114client: must fail: journalctl -u rs | grep -q 'reconnecting'6115worker1 # [ 111.515560] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:20Z level=info event=scheduler_connected addr=lb:500516116worker1 # [ 111.519597] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6117client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)6118lb: must succeed: curl -sf localhost:9901/clusters6119lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6120worker1 # [ 111.720603] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:20Z level=info event=scheduler_connected addr=lb:500516121worker1 # [ 111.724667] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6122worker1 # [ 111.925618] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:20Z level=info event=scheduler_connected addr=lb:500516123worker1 # [ 111.929611] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6124worker2 # [ 111.996645] nix-grpc-daemon[1109]: E0000 00:00:1789995561.059946 1154 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true6125worker2 # [ 112.001452] nix-grpc-daemon[1109]: E0000 00:00:1789995561.060095 1154 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6126worker1 # [ 112.132412] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:20Z level=info event=scheduler_connected addr=lb:500516127worker1 # [ 112.135557] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6128worker1 # [ 112.333812] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:21Z level=info event=scheduler_connected addr=lb:500516129worker1 # [ 112.337806] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6130worker1 # [ 112.538948] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:21Z level=info event=scheduler_connected addr=lb:500516131worker1 # [ 112.543231] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6132lb: must succeed: curl -sf localhost:9901/clusters6133lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6134worker1 # [ 112.744853] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:21Z level=info event=scheduler_connected addr=lb:500516135worker1 # [ 112.748690] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6136worker2 # [ 112.905825] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6137worker2 # [ 112.909747] systemd[1]: Stopped Nix worker-protocol over gRPC.6138worker2 # [ 112.911917] systemd[1]: nix-grpc-daemon.service: Consumed 1.449s CPU time over 36.189s wall clock time, 35.1M memory peak, 62.3K incoming IP traffic, 162.9K outgoing IP traffic.6139worker2 # [ 112.927228] systemd[1]: Starting scheduler-token.service...6140worker1 # [ 112.946814] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:21Z level=info event=scheduler_connected addr=lb:500516141worker1 # [ 112.950905] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6142worker2 # [ 113.005858] systemd[1]: scheduler-token.service: Deactivated successfully.6143worker2 # [ 113.006922] systemd[1]: Finished scheduler-token.service.6144worker2 # [ 113.008327] systemd[1]: scheduler-token.service: Consumed 22ms CPU time over 79ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6145worker2 # [ 113.013282] systemd[1]: Starting Nix worker-protocol over gRPC...6146worker2 # [ 113.177712] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:22Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6147worker2 # [ 113.180265] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:22Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6148worker1 # [ 113.153548] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:21Z level=info event=scheduler_connected addr=lb:500516149worker2 # [ 113.183775] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:22Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16150worker1 # [ 113.156696] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6151worker2 # [ 113.191062] nix-grpc-daemon[1398]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6152worker2 # [ 113.193096] nix-grpc-daemon[1398]: E0000 00:00:1789995562.254120 1398 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6153worker2 # [ 113.197594] nix-grpc-daemon[1398]: E0000 00:00:1789995562.259267 1398 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6154worker2 # [ 113.200729] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:22Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6155worker2 # [ 113.202652] systemd[1]: Started Nix worker-protocol over gRPC.6156lb # [ 113.177655] niks3-server[1174]: 2026/09/21 12:59:22 INFO lead: acquired remote=192.168.1.4:555006157worker2 # [ 113.209066] nix-grpc-daemon[1398]: E0000 00:00:1789995562.271524 1440 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6158worker2 # [ 113.212512] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:22Z level=info event=scheduler_take_over6159worker2 # [ 113.221455] nix-grpc-daemon[1398]: E0000 00:00:1789995562.284946 1440 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6160worker2 # [ 113.226055] nix-grpc-daemon[1398]: E0000 00:00:1789995562.287142 1440 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6161worker2 # [ 113.235430] nix-grpc-daemon[1398]: E0000 00:00:1789995562.298927 1440 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6162worker2 # [ 113.244376] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:22Z level=info event=scheduler_connected addr=lb:500516163worker2 # [ 113.245900] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6164worker1 # [ 113.354634] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:22Z level=info event=scheduler_connected addr=lb:500516165worker1 # [ 113.358786] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6166worker2 # [ 113.447371] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:22Z level=info event=scheduler_connected addr=lb:500516167worker2 # [ 113.451149] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6168worker1 # [ 113.559805] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:22Z level=info event=scheduler_connected addr=lb:500516169worker1 # [ 113.563914] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6170worker2 # [ 113.652294] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:22Z level=info event=scheduler_connected addr=lb:500516171worker2 # [ 113.656163] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6172lb: must succeed: curl -sf localhost:9901/clusters6173worker1 # [ 113.764914] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:22Z level=info event=scheduler_connected addr=lb:500516174worker1 # [ 113.768979] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6175lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6176worker2 # [ 113.857311] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:22Z level=info event=scheduler_connected addr=lb:500516177worker2 # [ 113.861236] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6178worker1 # [ 113.970238] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:22Z level=info event=scheduler_connected addr=lb:500516179worker2 # [ 114.002339] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:23Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:581366180worker2 # [ 114.006495] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:23Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06181worker2 # [ 114.062111] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:23Z level=info event=scheduler_connected addr=lb:500516182worker2 # [ 114.071132] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:23Z level=info event=oidc_jwks provider=mock keys=16183worker2 # [ 114.075239] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:23Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:581366184worker2 # [ 114.079079] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:23Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06185lb: must succeed: curl -sf localhost:9901/clusters6186lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6187worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6188worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6189(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 13.94 seconds)6190subtest: scheduler restart mid-build: build finishes, no double build6191client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid6192client # [ 115.007980] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid.6193client # Running as unit: mid.service; invocation ID: 3e4594c5896a493c91d956088d40bc946194client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag mid, in 0.07 seconds)6195client # [ 115.248039] nix[1238]: don't know how to build these paths:6196client # [ 115.249423] nix[1238]: /nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6197worker1 # [ 115.233381] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:459706198worker2 # [ 115.271524] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:24Z level=debug event=want client=1 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv cached=06199worker2 # [ 115.275944] nix-grpc-daemon[1398]: ts=2026-09-21T12:59:24Z level=debug event=assigned drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv worker=192.168.1.3:50051 clients=16200worker1 # [ 115.288913] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:459706201worker1 # [ 115.301708] nix-grpc-daemon[1435]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 59 ms (attempt 1/5)6202worker1 # [ 115.366544] nix-grpc-daemon[1435]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 35 ms (attempt 2/5)6203worker1 # [ 115.407478] nix-grpc-daemon[1435]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 165 ms (attempt 3/5)6204worker1 # [ 115.578755] nix-grpc-daemon[1435]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 186 ms (attempt 4/5)6205worker1 # [ 115.765583] nix-grpc-daemon[1435]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6206worker1 # [ 115.769395] nix-grpc-daemon[1435]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6207worker1 # [ 115.787206] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:459706208client # [ 115.810285] nix[1238]: copying 1 paths to 'lb:50051'...6209client # [ 115.813941] nix[1238]: copying path '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' to 'lb:50051'...6210worker1 # [ 115.800552] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:459706211worker1 # [ 115.812524] nix-daemon[849]: accepted connection from pid 1435, user nix-grpc-daemon (trusted)6212worker1 # [ 115.833582] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:24Z level=info event=niks3_push_started pid=15496213worker1 # [ 116.057233] nix-grpc-daemon[1549]: time=2026-09-21T12:59:24.899Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6214worker1 # [ 116.240607] nix-daemon[849]: accepted connection from pid 1562, user nix-grpc-daemon (trusted)6215worker1 # [ 116.254604] nix-daemon[849]: reaped child process 1569, status = succeeded6216worker1 # [ 116.435649] nix-daemon[849]: accepted connection from pid 1571, user nix-grpc-daemon (trusted)6217worker1 # [ 116.449080] nix-daemon[849]: reaped child process 1574, status = succeeded6218lb # [ 116.462311] niks3-server[1174]: 2026/09/21 12:59:25 INFO Received uploads request method=POST path=/api/pending_closures6219worker1 # [ 116.473555] nix-grpc-daemon[1549]: time=2026-09-21T12:59:25.316Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6220worker1 # [ 116.475139] nix-grpc-daemon[1549]: time=2026-09-21T12:59:25.318Z level=INFO msg="Uploading n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv (424B)"6221lb # [ 116.494576] niks3-server[1174]: 2026/09/21 12:59:25 INFO Registered completed upload object_key=nar/0pdyhjlvyi83rdx6rr15l7kj7a7hlcbqgllxm29xw3hvr3akprgq.nar.zst6222lb # [ 116.501796] niks3-server[1174]: 2026/09/21 12:59:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign6223lb # [ 116.507230] niks3-server[1174]: 2026/09/21 12:59:25 INFO Signed narinfos id=41 count=16224worker1 # [ 116.513636] nix-grpc-daemon[1549]: time=2026-09-21T12:59:25.356Z level=INFO msg="Uploading 1 narinfos"6225lb # [ 116.519495] niks3-server[1174]: 2026/09/21 12:59:25 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.ls6226lb # [ 116.528087] niks3-server[1174]: 2026/09/21 12:59:25 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete6227worker1 # [ 116.536120] nix-grpc-daemon[1549]: time=2026-09-21T12:59:25.379Z level=INFO msg="Upload complete. (475ms)"6228lb # [ 116.532752] niks3-server[1174]: 2026/09/21 12:59:25 INFO Completed upload id=416229worker1 # [ 116.538627] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45970 duration_s=0 paths=1 nar_bytes_in=4246230worker1 # [ 116.542664] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:459706231worker1 # [ 116.547596] nix-daemon[849]: accepted connection from pid 1435, user nix-grpc-daemon (trusted)6232lb # [ 116.548291] niks3-server[1174]: 2026/09/21 12:59:25 INFO Registered completed upload object_key=n23kf6h4k511p0qpzsydmqrbq2qn9g5w.narinfo6233worker1 # [ 116.554083] nix-daemon[849]: reaped child process 1547, status = succeeded6234worker1 # [ 116.559197] nix-daemon[849]: accepted connection from pid 1435, user nix-grpc-daemon (trusted)6235client # [ 116.573307] nix[1238]: building '/nix/store/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv' on lb:50051...6236lb # [ 118.357414] envoy[922]: conn peer=[2001:db8:1::1]:35560 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2960 tx=1430 ms=82776237lb # [ 118.361179] envoy[922]: conn peer=[2001:db8:1::4]:36950 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=29248 tx=1697 ms=343726238lb: must succeed: curl -sf localhost:9901/clusters6239lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6240lb: must succeed: curl -sf localhost:9901/clusters6241lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6242worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6243worker2 # [ 118.827881] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1398 (nix-grpc-daemon) on client request.6244worker2 # [ 118.839662] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6245lb # [ 118.815611] niks3-server[1174]: 2026/09/21 12:59:27 INFO lead: released remote=192.168.1.4:555006246client # [ 118.842127] nix[1238]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6247worker1 # [ 118.827894] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:27Z level=info event=scheduler_disconnected addr=lb:50051 error="Stream removed (RST_STREAM (Received RST_STREAM with error code 2))"6248worker2 # [ 118.861434] systemd[1]: Starting scheduler-token.service...6249worker2 # [ 118.869542] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6250worker2 # [ 118.871585] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6251worker2 # [ 118.874228] systemd[1]: nix-grpc-daemon.service: Consumed 153ms CPU time over 5.856s wall clock time, 13.3M memory peak, 10.1K incoming IP traffic, 16.8K outgoing IP traffic.6252worker2 # [ 118.932099] systemd[1]: scheduler-token.service: Deactivated successfully.6253worker2 # [ 118.933203] systemd[1]: Finished scheduler-token.service.6254worker2 # [ 118.934919] systemd[1]: scheduler-token.service: Consumed 16ms CPU time over 71ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6255worker2 # [ 118.937073] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6256worker2 # [ 118.941492] systemd[1]: Starting scheduler-token.service...6257worker2 # [ 119.002834] systemd[1]: scheduler-token.service: Deactivated successfully.6258worker2 # [ 119.003917] systemd[1]: Finished scheduler-token.service.6259worker2 # [ 119.005295] systemd[1]: scheduler-token.service: Consumed 16ms CPU time over 61ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6260worker2 # [ 119.010317] systemd[1]: Starting Nix worker-protocol over gRPC...6261worker1 # [ 119.028164] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:27Z level=info event=scheduler_connected addr=lb:500516262worker2 # [ 119.157313] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6263worker2 # [ 119.159257] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6264worker2 # [ 119.162938] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16265worker2 # [ 119.170066] nix-grpc-daemon[1488]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6266worker2 # [ 119.172095] nix-grpc-daemon[1488]: E0000 00:00:1789995568.233101 1488 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6267worker2 # [ 119.177746] nix-grpc-daemon[1488]: E0000 00:00:1789995568.238554 1488 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6268worker2 # [ 119.182725] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6269worker2 # [ 119.186134] systemd[1]: Started Nix worker-protocol over gRPC.6270worker2 # [ 119.191528] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=scheduler_take_over6271worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.40 seconds)6272client: must succeed: sleep 5.06273lb # [ 119.163219] niks3-server[1174]: 2026/09/21 12:59:28 INFO lead: acquired remote=192.168.1.4:555206274worker2 # [ 119.195063] nix-grpc-daemon[1488]: E0000 00:00:1789995568.258566 1529 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6275worker2 # [ 119.200825] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:581706276worker2 # [ 119.206209] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=16277worker2 # [ 119.215599] nix-grpc-daemon[1488]: E0000 00:00:1789995568.279038 1529 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6278worker2 # [ 119.220059] nix-grpc-daemon[1488]: E0000 00:00:1789995568.281532 1529 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6279worker2 # [ 119.230356] nix-grpc-daemon[1488]: E0000 00:00:1789995568.293818 1529 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6280worker2 # [ 119.239520] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=scheduler_connected addr=lb:500516281worker2 # [ 119.241145] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6282client # [ 119.344433] nix[1238]: scheduler at lb:50051: no healthy upstream, reconnecting6283worker2 # [ 119.442650] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=scheduler_connected addr=lb:500516284worker2 # [ 119.446503] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6285worker2 # [ 119.646957] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=scheduler_connected addr=lb:500516286worker2 # [ 119.648774] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6287worker2 # [ 119.849351] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=scheduler_connected addr=lb:500516288worker2 # [ 119.851858] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6289worker2 # [ 120.052926] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:29Z level=info event=scheduler_connected addr=lb:500516290worker2 # [ 120.056809] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6291worker2 # [ 120.260566] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:29Z level=info event=scheduler_connected addr=lb:500516292worker2 # [ 120.268175] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:29Z level=info event=oidc_jwks provider=mock keys=16293worker2 # [ 120.272490] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:29Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:581706294worker2 # [ 120.276437] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:29Z level=info event=worker_hello addr=192.168.1.4:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06295client: (finished: must succeed: sleep 5.0, in 5.05 seconds)6296client: waiting for success: ! systemctl is-active mid6297worker1 # [ 124.595345] nix-daemon[849]: reaped child process 1578, status = succeeded6298worker1 # [ 124.778659] nix-daemon[849]: accepted connection from pid 1615, user nix-grpc-daemon (trusted)6299worker1 # [ 124.792260] nix-daemon[849]: reaped child process 1618, status = succeeded6300lb # [ 124.807593] niks3-server[1174]: 2026/09/21 12:59:33 INFO Received uploads request method=POST path=/api/pending_closures6301worker1 # [ 124.820357] nix-grpc-daemon[1549]: time=2026-09-21T12:59:33.662Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6302worker1 # [ 124.821707] nix-grpc-daemon[1549]: time=2026-09-21T12:59:33.662Z level=INFO msg="Uploading cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj-slow-mid (120B)"6303lb # [ 124.844672] niks3-server[1174]: 2026/09/21 12:59:33 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6304lb # [ 124.854152] niks3-server[1174]: 2026/09/21 12:59:33 INFO Registered completed upload object_key=log/n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv6305lb # [ 124.863778] niks3-server[1174]: 2026/09/21 12:59:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign6306worker1 # [ 124.872200] nix-grpc-daemon[1549]: time=2026-09-21T12:59:33.714Z level=INFO msg="Uploading 1 narinfos"6307lb # [ 124.868570] niks3-server[1174]: 2026/09/21 12:59:33 INFO Signed narinfos id=42 count=16308lb # [ 124.873701] niks3-server[1174]: 2026/09/21 12:59:33 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.ls6309lb # [ 124.883719] niks3-server[1174]: 2026/09/21 12:59:33 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete6310lb # [ 124.889153] niks3-server[1174]: 2026/09/21 12:59:33 INFO Completed upload id=426311worker1 # [ 124.892512] nix-grpc-daemon[1549]: time=2026-09-21T12:59:33.735Z level=INFO msg="Upload complete. (315ms)"6312worker1 # [ 124.897265] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45970 duration_s=8 drv=n23kf6h4k511p0qpzsydmqrbq2qn9g5w-slow-mid.drv assign_id=1 outputs=16313lb # [ 124.900508] niks3-server[1174]: 2026/09/21 12:59:33 INFO Registered completed upload object_key=cz6wmnnj5y0qn84jrv9qb9sj0dv1iylj.narinfo6314worker1 # [ 124.909850] nix-daemon[849]: reaped child process 1576, status = succeeded6315client # [ 124.951932] systemd[1]: mid.service: Deactivated successfully.6316client # [ 124.955497] systemd[1]: mid.service: Consumed 129ms CPU time over 9.946s wall clock time, 41.4M memory peak, 7.9K incoming IP traffic, 9K outgoing IP traffic.6317client: (finished: waiting for success: ! systemctl is-active mid, in 1.11 seconds)6318client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6319client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.04 seconds)6320(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.68 seconds)6321subtest: interrupting the client stops the build on the worker6322client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr6323client # [ 125.661975] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr.6324client # Running as unit: intr.service; invocation ID: 6962470666764ca3a3c6f06e63472cc66325client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag intr, in 0.04 seconds)6326worker1 # [ 125.854332] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:459706327client # [ 125.874453] nix[1274]: don't know how to build these paths:6328client # [ 125.875352] nix[1274]: /nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6329worker2 # [ 125.898298] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:34Z level=debug event=want client=2 drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv cached=06330worker2 # [ 125.902881] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:34Z level=debug event=assigned drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv worker=192.168.1.4:50051 clients=16331worker2 # [ 125.943151] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436746332worker2 # [ 125.954287] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6333worker2 # [ 125.970230] nix-grpc-daemon[1488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 90 ms (attempt 1/5)6334worker2 # [ 126.065958] nix-grpc-daemon[1488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 193 ms (attempt 2/5)6335worker2 # [ 126.264888] nix-grpc-daemon[1488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 118 ms (attempt 3/5)6336worker2 # [ 126.388705] nix-grpc-daemon[1488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 258 ms (attempt 4/5)6337worker2 # [ 126.652708] nix-grpc-daemon[1488]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6338worker2 # [ 126.656274] nix-grpc-daemon[1488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6339worker2 # [ 126.672803] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436746340client # [ 126.671128] nix[1274]: copying 1 paths to 'lb:50051'...6341client # [ 126.674180] nix[1274]: copying path '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' to 'lb:50051'...6342worker2 # [ 126.687675] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436746343worker2 # [ 126.695577] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6344worker2 # [ 126.715823] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:35Z level=info event=niks3_push_started pid=15626345worker2 # [ 126.874597] nix-grpc-daemon[1562]: time=2026-09-21T12:59:35.937Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker2.key ca=""6346worker2 # [ 127.012126] nix-daemon[873]: accepted connection from pid 1575, user nix-grpc-daemon (trusted)6347worker2 # [ 127.022449] nix-daemon[873]: reaped child process 1582, status = succeeded6348worker2 # [ 127.182637] nix-daemon[873]: accepted connection from pid 1584, user nix-grpc-daemon (trusted)6349worker2 # [ 127.194395] nix-daemon[873]: reaped child process 1587, status = succeeded6350lb # [ 127.183291] niks3-server[1174]: 2026/09/21 12:59:36 INFO Received uploads request method=POST path=/api/pending_closures6351worker2 # [ 127.225659] nix-grpc-daemon[1562]: time=2026-09-21T12:59:36.288Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6352worker2 # [ 127.227156] nix-grpc-daemon[1562]: time=2026-09-21T12:59:36.288Z level=INFO msg="Uploading x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv (424B)"6353lb # [ 127.222597] niks3-server[1174]: 2026/09/21 12:59:36 INFO Registered completed upload object_key=nar/0dw6lfdicxarqmcw4m9j94irjhgkdvc1nww51zwj454yj03v5iri.nar.zst6354lb # [ 127.229584] niks3-server[1174]: 2026/09/21 12:59:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6355lb # [ 127.238355] niks3-server[1174]: 2026/09/21 12:59:36 INFO Signed narinfos id=43 count=16356worker2 # [ 127.271255] nix-grpc-daemon[1562]: time=2026-09-21T12:59:36.334Z level=INFO msg="Uploading 1 narinfos"6357lb # [ 127.244707] niks3-server[1174]: 2026/09/21 12:59:36 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.ls6358lb # [ 127.255213] niks3-server[1174]: 2026/09/21 12:59:36 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6359lb # [ 127.261817] niks3-server[1174]: 2026/09/21 12:59:36 INFO Registered completed upload object_key=x43jxvh618x0k4fbhc2wyxhr1lymdssr.narinfo6360lb # [ 127.264893] niks3-server[1174]: 2026/09/21 12:59:36 INFO Completed upload id=436361worker2 # [ 127.294596] nix-grpc-daemon[1562]: time=2026-09-21T12:59:36.358Z level=INFO msg="Upload complete. (416ms)"6362worker2 # [ 127.300514] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43674 duration_s=0 paths=1 nar_bytes_in=4246363worker2 # [ 127.305350] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436746364worker2 # [ 127.313479] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6365worker2 # [ 127.319866] nix-daemon[873]: reaped child process 1560, status = succeeded6366worker2 # [ 127.324768] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6367client # [ 127.312743] nix[1274]: building '/nix/store/x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv' on lb:50051...6368lb # [ 128.357972] envoy[922]: conn peer=[2001:db8:1::4]:52904 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=4422 tx=385 ms=56136369lb # [ 128.361777] envoy[922]: conn peer=[2001:db8:1::1]:52912 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3135 tx=1589 ms=96776370client: must succeed: systemctl kill -s INT intr6371client # [ 129.237448] systemd[1]: intr.service: Sent signal SIGINT to main process 1274 (nix) on client request.6372worker2 # [ 129.251778] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:38Z level=info event=revoked_running drv=x43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv6373client # [ 129.244556] nix[1274]: shutting downx43jxvh618x0k4fbhc2wyxhr1lymdssr-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6374client # [ 129.250101] nix[1274]: error: interrupted by the user6375client: (finished: must succeed: systemctl kill -s INT intr, in 0.06 seconds)6376client # [ 129.291333] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6377client # [ 129.292780] systemd[1]: intr.service: Failed with result 'exit-code'.6378client # [ 129.294684] systemd[1]: intr.service: Consumed 117ms CPU time over 3.632s wall clock time, 41.4M memory peak, 6.4K incoming IP traffic, 8.4K outgoing IP traffic.6379worker2 # [ 129.346266] nix-daemon[873]: reaped child process 1589, status = succeeded6380worker2 # [ 129.351515] nix-daemon[1591]: unexpected Nix daemon error: error: interrupted by the user6381worker2 # [ 129.356557] nix-daemon[873]: reaped child process 1591, status = failed with exit code 16382(finished: subtest: interrupting the client stops the build on the worker, in 3.74 seconds)6383subtest: a deploy mid-build drains: the build finishes, the new generation takes over6384client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw6385client # [ 129.413644] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key" --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw.6386client # Running as unit: sw.service; invocation ID: 11728fa6c2b1474e9164fa6b44b1d5486387client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/yggfr0llgkhpsm4k92sg65ggxk04755p-slow.nix --argstr tag sw, in 0.06 seconds)6388client # [ 129.632669] nix[1295]: don't know how to build these paths:6389client # [ 129.633566] nix[1295]: /nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6390worker1 # [ 129.616087] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:38Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:459706391worker2 # [ 129.653273] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:38Z level=debug event=want client=3 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv cached=06392worker2 # [ 129.657190] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:38Z level=debug event=assigned drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv worker=192.168.1.3:50051 clients=16393worker1 # [ 129.680501] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:459706394worker1 # [ 129.694417] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:459706395client # [ 129.719196] nix[1295]: copying 1 paths to 'lb:50051'...6396worker1 # [ 129.706913] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:459706397client # [ 129.722161] nix[1295]: copying path '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' to 'lb:50051'...6398worker1 # [ 129.721275] nix-daemon[849]: accepted connection from pid 1435, user nix-grpc-daemon (trusted)6399worker1 # [ 129.930236] nix-daemon[849]: accepted connection from pid 1646, user nix-grpc-daemon (trusted)6400worker1 # [ 129.944649] nix-daemon[849]: reaped child process 1649, status = succeeded6401worker1 # [ 130.144134] nix-daemon[849]: accepted connection from pid 1651, user nix-grpc-daemon (trusted)6402worker1 # [ 130.158303] nix-daemon[849]: reaped child process 1654, status = succeeded6403lb # [ 130.174346] niks3-server[1174]: 2026/09/21 12:59:39 INFO Received uploads request method=POST path=/api/pending_closures6404worker1 # [ 130.187644] nix-grpc-daemon[1549]: time=2026-09-21T12:59:39.030Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6405worker1 # [ 130.189250] nix-grpc-daemon[1549]: time=2026-09-21T12:59:39.032Z level=INFO msg="Uploading q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv (416B)"6406lb # [ 130.206165] niks3-server[1174]: 2026/09/21 12:59:39 INFO Registered completed upload object_key=nar/1rby34xp94wipnqa4abnh2771jnqp34zv82lymaabrhj07vmwiix.nar.zst6407lb # [ 130.219679] niks3-server[1174]: 2026/09/21 12:59:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6408lb # [ 130.225831] niks3-server[1174]: 2026/09/21 12:59:39 INFO Signed narinfos id=44 count=16409worker1 # [ 130.232103] nix-grpc-daemon[1549]: time=2026-09-21T12:59:39.075Z level=INFO msg="Uploading 1 narinfos"6410lb # [ 130.229280] niks3-server[1174]: 2026/09/21 12:59:39 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.ls6411lb # [ 130.242670] niks3-server[1174]: 2026/09/21 12:59:39 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6412lb # [ 130.248218] niks3-server[1174]: 2026/09/21 12:59:39 INFO Completed upload id=446413worker1 # [ 130.251517] nix-grpc-daemon[1549]: time=2026-09-21T12:59:39.094Z level=INFO msg="Upload complete. (516ms)"6414lb # [ 130.252557] niks3-server[1174]: 2026/09/21 12:59:39 INFO Registered completed upload object_key=q9pl41yvj7mi4fqnyiarzy5py0107h7b.narinfo6415worker1 # [ 130.255458] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45970 duration_s=0 paths=1 nar_bytes_in=4166416worker1 # [ 130.261138] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:459706417worker1 # [ 130.269504] nix-daemon[849]: accepted connection from pid 1435, user nix-grpc-daemon (trusted)6418worker1 # [ 130.277369] nix-daemon[849]: reaped child process 1644, status = succeeded6419worker1 # [ 130.282559] nix-daemon[849]: accepted connection from pid 1435, user nix-grpc-daemon (trusted)6420client # [ 130.295982] nix[1295]: building '/nix/store/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv' on lb:50051...6421worker1: must succeed: systemctl reload nix-grpc-daemon.service6422worker1 # [ 133.117302] systemd[1]: Reloading Nix worker-protocol over gRPC...6423worker1 # [ 133.166071] systemd[1]: Reloaded Nix worker-protocol over gRPC.6424worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.09 seconds)6425worker1: must succeed: pgrep -f 'read -t [9]0 x'6426worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)6427client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&26428worker1 # [ 133.473301] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:42Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:459706429client # don't know how to build these paths:6430client # /nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6431worker2 # [ 133.507589] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436746432worker2 # [ 133.529464] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:42Z level=debug event=want client=4 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv cached=06433worker2 # [ 133.534614] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:42Z level=debug event=assigned drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv worker=192.168.1.4:50051 clients=16434worker2 # [ 133.579222] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436746435worker2 # [ 133.591257] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436746436client # copying 1 paths to 'lb:50051'...6437worker2 # [ 133.603503] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436746438client # copying path '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' to 'lb:50051'...6439worker2 # [ 133.624198] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6440worker1 # [ 133.702052] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:42Z level=info event=draining6441worker2 # [ 133.809114] nix-daemon[873]: accepted connection from pid 1646, user nix-grpc-daemon (trusted)6442worker2 # [ 133.820583] nix-daemon[873]: reaped child process 1649, status = succeeded6443worker2 # [ 133.996113] nix-daemon[873]: accepted connection from pid 1651, user nix-grpc-daemon (trusted)6444worker2 # [ 134.008111] nix-daemon[873]: reaped child process 1654, status = succeeded6445lb # [ 133.997684] niks3-server[1174]: 2026/09/21 12:59:42 INFO Received uploads request method=POST path=/api/pending_closures6446worker2 # [ 134.037252] nix-grpc-daemon[1562]: time=2026-09-21T12:59:43.100Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6447worker2 # [ 134.038870] nix-grpc-daemon[1562]: time=2026-09-21T12:59:43.100Z level=INFO msg="Uploading s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv (528B)"6448lb # [ 134.023156] niks3-server[1174]: 2026/09/21 12:59:42 INFO Registered completed upload object_key=nar/0n1m04akn0svzkx0szkj4q7kvaw06fa9yx6k2jibkbkkv85mm55n.nar.zst6449lb # [ 134.035431] niks3-server[1174]: 2026/09/21 12:59:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6450lb # [ 134.040486] niks3-server[1174]: 2026/09/21 12:59:42 INFO Signed narinfos id=45 count=16451worker2 # [ 134.072957] nix-grpc-daemon[1562]: time=2026-09-21T12:59:43.136Z level=INFO msg="Uploading 1 narinfos"6452lb # [ 134.047899] niks3-server[1174]: 2026/09/21 12:59:42 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.ls6453lb # [ 134.053055] niks3-server[1174]: 2026/09/21 12:59:42 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6454lb # [ 134.057248] niks3-server[1174]: 2026/09/21 12:59:42 INFO Completed upload id=456455worker2 # [ 134.086671] nix-grpc-daemon[1562]: time=2026-09-21T12:59:43.150Z level=INFO msg="Upload complete. (451ms)"6456lb # [ 134.058897] niks3-server[1174]: 2026/09/21 12:59:42 INFO Registered completed upload object_key=s7lncxagqa332zn7mhwmk3ry9d83sbvq.narinfo6457worker2 # [ 134.089473] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43674 duration_s=0 paths=1 nar_bytes_in=5286458worker2 # [ 134.094456] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436746459worker2 # [ 134.102949] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6460worker2 # [ 134.105241] nix-daemon[873]: reaped child process 1644, status = succeeded6461worker2 # [ 134.110561] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6462client # building '/nix/store/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv' on lb:50051...6463client # farm-a> worker2: building s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6464client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6465client # farm-a> LOG-farm-a-during-drain6466worker2 # [ 135.324743] nix-daemon[873]: reaped child process 1658, status = succeeded6467worker2 # [ 135.474920] nix-daemon[873]: accepted connection from pid 1674, user nix-grpc-daemon (trusted)6468worker2 # [ 135.486857] nix-daemon[873]: reaped child process 1677, status = succeeded6469lb # [ 135.474221] niks3-server[1174]: 2026/09/21 12:59:44 INFO Received uploads request method=POST path=/api/pending_closures6470worker2 # [ 135.512663] nix-grpc-daemon[1562]: time=2026-09-21T12:59:44.575Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6471worker2 # [ 135.514316] nix-grpc-daemon[1562]: time=2026-09-21T12:59:44.577Z level=INFO msg="Uploading zhl4afvdy7vs34m6ngc3z1xk04qgpcfs-farm-a (136B)"6472lb # [ 135.519833] niks3-server[1174]: 2026/09/21 12:59:44 INFO Registered completed upload object_key=log/s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv6473lb # [ 135.527413] niks3-server[1174]: 2026/09/21 12:59:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6474lb # [ 135.535385] niks3-server[1174]: 2026/09/21 12:59:44 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6475lb # [ 135.542389] niks3-server[1174]: 2026/09/21 12:59:44 INFO Signed narinfos id=46 count=16476worker2 # [ 135.572220] nix-grpc-daemon[1562]: time=2026-09-21T12:59:44.635Z level=INFO msg="Uploading 1 narinfos"6477lb # [ 135.556785] niks3-server[1174]: 2026/09/21 12:59:44 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6478lb # [ 135.562214] niks3-server[1174]: 2026/09/21 12:59:44 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.ls6479lb # [ 135.565110] niks3-server[1174]: 2026/09/21 12:59:44 INFO Completed upload id=466480worker2 # [ 135.594806] nix-grpc-daemon[1562]: time=2026-09-21T12:59:44.658Z level=INFO msg="Upload complete. (276ms)"6481lb # [ 135.567324] niks3-server[1174]: 2026/09/21 12:59:44 INFO Registered completed upload object_key=zhl4afvdy7vs34m6ngc3z1xk04qgpcfs.narinfo6482worker2 # [ 135.602634] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43674 duration_s=1 drv=s7lncxagqa332zn7mhwmk3ry9d83sbvq-farm-a.drv assign_id=3 outputs=16483worker2 # [ 135.612613] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:44Z level=debug event=want client=4 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv cached=06484worker2 # [ 135.616453] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:44Z level=debug event=assigned drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv worker=192.168.1.4:50051 clients=16485worker2 # [ 135.625088] nix-daemon[873]: reaped child process 1656, status = succeeded6486worker2 # [ 135.657335] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436746487worker2 # [ 135.669873] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436746488worker2 # [ 135.684814] nix-grpc-daemon[1562]: time=2026-09-21T12:59:44.748Z level=INFO msg="All 1 paths already cached"6489client # copying 1 paths to 'lb:50051'...6490worker2 # [ 135.693248] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436746491client # copying path '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' to 'lb:50051'...6492worker2 # [ 135.712145] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6493worker2 # [ 135.888751] nix-daemon[873]: accepted connection from pid 1683, user nix-grpc-daemon (trusted)6494worker2 # [ 135.901269] nix-daemon[873]: reaped child process 1686, status = succeeded6495worker2 # [ 136.071215] nix-daemon[873]: accepted connection from pid 1688, user nix-grpc-daemon (trusted)6496worker2 # [ 136.082817] nix-daemon[873]: reaped child process 1691, status = succeeded6497lb # [ 136.071153] niks3-server[1174]: 2026/09/21 12:59:44 INFO Received uploads request method=POST path=/api/pending_closures6498worker2 # [ 136.108633] nix-grpc-daemon[1562]: time=2026-09-21T12:59:45.172Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6499worker2 # [ 136.110768] nix-grpc-daemon[1562]: time=2026-09-21T12:59:45.173Z level=INFO msg="Uploading pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv (640B)"6500lb # [ 136.099559] niks3-server[1174]: 2026/09/21 12:59:45 INFO Registered completed upload object_key=nar/111ql9hf3mg62sp5fwspqwm8cl88198mlcc1ij8flm7l424sz63r.nar.zst6501lb # [ 136.109364] niks3-server[1174]: 2026/09/21 12:59:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6502lb # [ 136.113625] niks3-server[1174]: 2026/09/21 12:59:45 INFO Signed narinfos id=47 count=16503worker2 # [ 136.146187] nix-grpc-daemon[1562]: time=2026-09-21T12:59:45.209Z level=INFO msg="Uploading 1 narinfos"6504lb # [ 136.133291] niks3-server[1174]: 2026/09/21 12:59:45 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.ls6505lb # [ 136.136259] niks3-server[1174]: 2026/09/21 12:59:45 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6506lb # [ 136.140496] niks3-server[1174]: 2026/09/21 12:59:45 INFO Completed upload id=476507worker2 # [ 136.170196] nix-grpc-daemon[1562]: time=2026-09-21T12:59:45.233Z level=INFO msg="Upload complete. (446ms)"6508lb # [ 136.142176] niks3-server[1174]: 2026/09/21 12:59:45 INFO Registered completed upload object_key=pmcfhwqsl1hiyc8gjd850vixdadvskwn.narinfo6509worker2 # [ 136.175147] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43674 duration_s=0 paths=1 nar_bytes_in=6406510worker2 # [ 136.179666] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436746511worker2 # [ 136.187957] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6512worker2 # [ 136.192943] nix-daemon[873]: reaped child process 1681, status = succeeded6513worker2 # [ 136.197761] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6514client # building '/nix/store/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv' on lb:50051...6515client # farm-b> worker2: building pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6516client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6517client # farm-b> LOG-farm-b-during-drain6518worker2 # [ 137.622665] nix-daemon[873]: reaped child process 1695, status = succeeded6519worker2 # [ 137.788167] nix-daemon[873]: accepted connection from pid 1711, user nix-grpc-daemon (trusted)6520worker2 # [ 137.799981] nix-daemon[873]: reaped child process 1714, status = succeeded6521lb # [ 137.789376] niks3-server[1174]: 2026/09/21 12:59:46 INFO Received uploads request method=POST path=/api/pending_closures6522worker2 # [ 137.830735] nix-grpc-daemon[1562]: time=2026-09-21T12:59:46.894Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6523worker2 # [ 137.832795] nix-grpc-daemon[1562]: time=2026-09-21T12:59:46.895Z level=INFO msg="Uploading l458nfm9f1f1zyghsy09ijx02qwk7v65-farm-b (184B)"6524lb # [ 137.830724] niks3-server[1174]: 2026/09/21 12:59:46 INFO Registered completed upload object_key=log/pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv6525lb # [ 137.837694] niks3-server[1174]: 2026/09/21 12:59:46 INFO Registered completed upload object_key=nar/0m50ra4cd4i97vcbcrdsw7xn8f0b3zc5pdyr2ab8kmp3hscd8j80.nar.zst6526lb # [ 137.851259] niks3-server[1174]: 2026/09/21 12:59:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6527lb # [ 137.856509] niks3-server[1174]: 2026/09/21 12:59:46 INFO Signed narinfos id=48 count=16528worker2 # [ 137.889825] nix-grpc-daemon[1562]: time=2026-09-21T12:59:46.952Z level=INFO msg="Uploading 1 narinfos"6529lb # [ 137.861653] niks3-server[1174]: 2026/09/21 12:59:46 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.ls6530lb # [ 137.872220] niks3-server[1174]: 2026/09/21 12:59:46 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6531lb # [ 137.878721] niks3-server[1174]: 2026/09/21 12:59:46 INFO Completed upload id=486532worker2 # [ 137.908338] nix-grpc-daemon[1562]: time=2026-09-21T12:59:46.971Z level=INFO msg="Upload complete. (302ms)"6533lb # [ 137.881112] niks3-server[1174]: 2026/09/21 12:59:46 INFO Registered completed upload object_key=l458nfm9f1f1zyghsy09ijx02qwk7v65.narinfo6534worker2 # [ 137.915933] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43674 duration_s=1 drv=pmcfhwqsl1hiyc8gjd850vixdadvskwn-farm-b.drv assign_id=4 outputs=16535worker2 # [ 137.927102] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:46Z level=debug event=want client=4 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv cached=06536worker2 # [ 137.932251] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:46Z level=debug event=assigned drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv worker=192.168.1.4:50051 clients=16537worker2 # [ 137.941053] nix-daemon[873]: reaped child process 1693, status = succeeded6538worker2 # [ 137.970080] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436746539worker2 # [ 137.982661] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436746540worker2 # [ 137.995201] nix-grpc-daemon[1562]: time=2026-09-21T12:59:47.058Z level=INFO msg="All 2 paths already cached"6541client # copying 1 paths to 'lb:50051'...6542worker2 # [ 138.003522] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436746543client # copying path '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' to 'lb:50051'...6544worker2 # [ 138.023162] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6545worker2 # [ 138.199570] nix-daemon[873]: accepted connection from pid 1720, user nix-grpc-daemon (trusted)6546worker2 # [ 138.211500] nix-daemon[873]: reaped child process 1723, status = succeeded6547worker2 # [ 138.381885] nix-daemon[873]: accepted connection from pid 1725, user nix-grpc-daemon (trusted)6548lb # [ 138.357896] envoy[922]: conn peer=[2001:db8:1::1]:44268 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2993 tx=829 ms=33926549worker2 # [ 138.393630] nix-daemon[873]: reaped child process 1728, status = succeeded6550lb # [ 138.381378] niks3-server[1174]: 2026/09/21 12:59:47 INFO Received uploads request method=POST path=/api/pending_closures6551worker2 # [ 138.418308] nix-grpc-daemon[1562]: time=2026-09-21T12:59:47.481Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6552worker2 # [ 138.419901] nix-grpc-daemon[1562]: time=2026-09-21T12:59:47.483Z level=INFO msg="Uploading w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv (768B)"6553lb # [ 138.409103] niks3-server[1174]: 2026/09/21 12:59:47 INFO Registered completed upload object_key=nar/1crrgw2l7pdmf28ls634k8y85m0pc1v13pxxp4rd6yg5mk3kr5fs.nar.zst6554lb # [ 138.420740] niks3-server[1174]: 2026/09/21 12:59:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6555worker2 # [ 138.455213] nix-grpc-daemon[1562]: time=2026-09-21T12:59:47.518Z level=INFO msg="Uploading 1 narinfos"6556lb # [ 138.426123] niks3-server[1174]: 2026/09/21 12:59:47 INFO Signed narinfos id=49 count=16557lb # [ 138.432170] niks3-server[1174]: 2026/09/21 12:59:47 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.ls6558lb # [ 138.441335] niks3-server[1174]: 2026/09/21 12:59:47 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6559lb # [ 138.447389] niks3-server[1174]: 2026/09/21 12:59:47 INFO Completed upload id=496560worker2 # [ 138.476960] nix-grpc-daemon[1562]: time=2026-09-21T12:59:47.540Z level=INFO msg="Upload complete. (441ms)"6561lb # [ 138.452175] niks3-server[1174]: 2026/09/21 12:59:47 INFO Registered completed upload object_key=w4kjbckcz25f2jivly0r4lbq18qf474f.narinfo6562worker2 # [ 138.481128] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43674 duration_s=0 paths=1 nar_bytes_in=7686563worker2 # [ 138.485921] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436746564worker2 # [ 138.496103] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6565worker2 # [ 138.499389] nix-daemon[873]: reaped child process 1718, status = succeeded6566worker2 # [ 138.504483] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)6567client # building '/nix/store/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv' on lb:50051...6568client # farm-top> worker2: building w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6569client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6570client # farm-top> LOG-farm-top-during-drain6571worker2 # [ 139.528728] nix-daemon[873]: reaped child process 1732, status = succeeded6572worker2 # [ 139.698347] nix-daemon[873]: accepted connection from pid 1748, user nix-grpc-daemon (trusted)6573worker2 # [ 139.710250] nix-daemon[873]: reaped child process 1751, status = succeeded6574lb # [ 139.698922] niks3-server[1174]: 2026/09/21 12:59:48 INFO Received uploads request method=POST path=/api/pending_closures6575worker2 # [ 139.737788] nix-grpc-daemon[1562]: time=2026-09-21T12:59:48.801Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6576worker2 # [ 139.739619] nix-grpc-daemon[1562]: time=2026-09-21T12:59:48.802Z level=INFO msg="Uploading 08ap9ki0d5gixmyr2sac4hj06j9jsqhg-farm-top (240B)"6577lb # [ 139.734794] niks3-server[1174]: 2026/09/21 12:59:48 INFO Registered completed upload object_key=log/w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv6578lb # [ 139.741954] niks3-server[1174]: 2026/09/21 12:59:48 INFO Registered completed upload object_key=nar/01f0abadm30zaav33qck3x7ypr9nn6rqh2vbrvih4gyb8c26x9in.nar.zst6579lb # [ 139.753217] niks3-server[1174]: 2026/09/21 12:59:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6580lb # [ 139.759052] niks3-server[1174]: 2026/09/21 12:59:48 INFO Signed narinfos id=50 count=16581worker2 # [ 139.791052] nix-grpc-daemon[1562]: time=2026-09-21T12:59:48.854Z level=INFO msg="Uploading 1 narinfos"6582lb # [ 139.762767] niks3-server[1174]: 2026/09/21 12:59:48 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.ls6583lb # [ 139.767362] niks3-server[1174]: 2026/09/21 12:59:48 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6584lb # [ 139.771038] niks3-server[1174]: 2026/09/21 12:59:48 INFO Completed upload id=506585lb # [ 139.772880] niks3-server[1174]: 2026/09/21 12:59:48 INFO Registered completed upload object_key=08ap9ki0d5gixmyr2sac4hj06j9jsqhg.narinfo6586worker2 # [ 139.801993] nix-grpc-daemon[1562]: time=2026-09-21T12:59:48.865Z level=INFO msg="Upload complete. (290ms)"6587worker2 # [ 139.808414] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43674 duration_s=1 drv=w4kjbckcz25f2jivly0r4lbq18qf474f-farm-top.drv assign_id=5 outputs=16588worker2 # [ 139.819120] nix-daemon[873]: reaped child process 1730, status = succeeded6589client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain >&2, in 6.61 seconds)6590client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath6591client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag during-drain outPath, in 0.22 seconds)6592worker1: must succeed: pkill -f 'read -t [9]0 x'6593worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.06 seconds)6594client: waiting for success: ! systemctl is-active sw6595worker1 # [ 140.321856] nix-daemon[849]: reaped child process 1658, status = succeeded6596worker1 # [ 140.515102] nix-daemon[849]: accepted connection from pid 1704, user nix-grpc-daemon (trusted)6597worker1 # [ 140.527607] nix-daemon[849]: reaped child process 1707, status = succeeded6598lb # [ 140.544417] niks3-server[1174]: 2026/09/21 12:59:49 INFO Received uploads request method=POST path=/api/pending_closures6599worker1 # [ 140.556393] nix-grpc-daemon[1549]: time=2026-09-21T12:59:49.399Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6600worker1 # [ 140.558734] nix-grpc-daemon[1549]: time=2026-09-21T12:59:49.401Z level=INFO msg="Uploading mlk55snqrikr02pwsxjamphvl61rvl0r-slow-sw (120B)"6601lb # [ 140.576787] niks3-server[1174]: 2026/09/21 12:59:49 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6602lb # [ 140.588139] niks3-server[1174]: 2026/09/21 12:59:49 INFO Registered completed upload object_key=log/q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv6603lb # [ 140.598109] niks3-server[1174]: 2026/09/21 12:59:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6604worker1 # [ 140.606049] nix-grpc-daemon[1549]: time=2026-09-21T12:59:49.449Z level=INFO msg="Uploading 1 narinfos"6605lb # [ 140.603070] niks3-server[1174]: 2026/09/21 12:59:49 INFO Signed narinfos id=51 count=16606lb # [ 140.608472] niks3-server[1174]: 2026/09/21 12:59:49 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.ls6607lb # [ 140.617245] niks3-server[1174]: 2026/09/21 12:59:49 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6608lb # [ 140.622096] niks3-server[1174]: 2026/09/21 12:59:49 INFO Completed upload id=516609worker1 # [ 140.626982] nix-grpc-daemon[1549]: time=2026-09-21T12:59:49.470Z level=INFO msg="Upload complete. (325ms)"6610lb # [ 140.625298] niks3-server[1174]: 2026/09/21 12:59:49 INFO Registered completed upload object_key=mlk55snqrikr02pwsxjamphvl61rvl0r.narinfo6611worker1 # [ 140.633681] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45970 duration_s=10 drv=q9pl41yvj7mi4fqnyiarzy5py0107h7b-slow-sw.drv assign_id=2 outputs=16612worker1 # [ 140.647271] nix-daemon[849]: reaped child process 1656, status = succeeded6613client # [ 140.687482] systemd[1]: sw.service: Deactivated successfully.6614client # [ 140.691748] systemd[1]: sw.service: Consumed 122ms CPU time over 11.275s wall clock time, 41.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.6615worker1 # [ 140.704956] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:49Z level=info event=signal_exit6616client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds)6617client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6618client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.04 seconds)6619worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6620worker2 # [ 143.967779] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:53Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:581706621worker2 # [ 143.971904] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:53Z level=info event=worker_gone addr=192.168.1.3:500516622worker1 # [ 143.942803] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:52Z level=info event=scheduler_restarting6623worker1 # [ 143.956395] nix-grpc-daemon[1435]: ts=2026-09-21T12:59:52Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6624worker1 # [ 143.974206] nix-daemon[849]: reaped child process 1494, status = succeeded6625worker1 # [ 144.024961] nix-grpc-daemon[1435]: E0000 00:00:1789995592.868153 1480 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true6626worker1 # [ 144.029839] nix-grpc-daemon[1435]: E0000 00:00:1789995592.868295 1480 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6627worker1 # [ 145.086747] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6628worker1 # [ 145.090694] systemd[1]: nix-grpc-daemon.service: Consumed 1.003s CPU time over 44.603s wall clock time, 33.1M memory peak, 27.3K incoming IP traffic, 40.9K outgoing IP traffic.6629worker1 # [ 145.107519] systemd[1]: Starting Nix worker-protocol over gRPC...6630worker1 # [ 145.297885] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:54Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6631worker1 # [ 145.300295] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:54Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6632worker1 # [ 145.304408] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:54Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16633worker1 # [ 145.312065] nix-grpc-daemon[1721]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6634worker1 # [ 145.314186] nix-grpc-daemon[1721]: E0000 00:00:1789995594.155136 1721 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6635worker1 # [ 145.318110] nix-grpc-daemon[1721]: E0000 00:00:1789995594.160306 1721 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6636worker1 # [ 145.320570] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:54Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6637worker1 # [ 145.326208] systemd[1]: Started Nix worker-protocol over gRPC.6638worker1 # [ 145.333077] nix-grpc-daemon[1721]: E0000 00:00:1789995594.175319 1761 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6639worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 4.10 seconds)6640lb: waiting until exactly no endpoints of x86_64-linux are unhealthy6641lb: must succeed: curl -sf localhost:9901/clusters6642worker1 # [ 145.347127] nix-grpc-daemon[1721]: E0000 00:00:1789995594.188813 1761 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6643worker1 # [ 145.350156] nix-grpc-daemon[1721]: E0000 00:00:1789995594.188961 1761 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6644worker1 # [ 145.360055] nix-grpc-daemon[1721]: E0000 00:00:1789995594.203439 1761 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6645worker1 # [ 145.370820] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:54Z level=info event=scheduler_connected addr=lb:500516646worker2 # [ 145.400355] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:54Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:581706647worker2 # [ 145.404598] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:54Z level=info event=worker_hello addr=192.168.1.3:50051 systems=x86_64-linux,i686-linux,x86_64-v1-linux,x86_64-v2-linux,x86_64-v3-linux,x86_64-v4-linux max_jobs=1 running=06648lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6649lb: must succeed: curl -sf localhost:9901/clusters6650lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6651lb: (finished: waiting until exactly no endpoints of x86_64-linux are unhealthy, in 1.16 seconds)6652lb: must succeed: curl -sf localhost:9901/clusters6653lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6654worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6655worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6656(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 17.25 seconds)6657subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6658client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16659worker2 # [ 146.881216] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:55Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:436746660worker2 # [ 146.885320] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436746661worker2 # [ 146.920661] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:55Z level=debug event=want client=5 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv cached=06662worker2 # [ 146.922516] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:55Z level=debug event=assigned drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv worker=192.168.1.3:50051 clients=16663worker2 # [ 146.924392] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:55Z level=debug event=want client=5 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv cached=06664worker2 # [ 146.925959] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:55Z level=debug event=want client=5 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv cached=06665worker1 # [ 146.943728] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386666worker1 # [ 146.955865] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6667worker1 # [ 146.973870] nix-grpc-daemon[1721]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 88 ms (attempt 1/5)6668worker1 # [ 147.067985] nix-grpc-daemon[1721]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 100 ms (attempt 2/5)6669worker1 # [ 147.173780] nix-grpc-daemon[1721]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 258 ms (attempt 3/5)6670worker1 # [ 147.437806] nix-grpc-daemon[1721]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 346 ms (attempt 4/5)6671worker1 # [ 147.790059] nix-grpc-daemon[1721]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6672worker1 # [ 147.794167] nix-grpc-daemon[1721]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6673worker1 # [ 147.811534] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572386674worker1 # [ 147.824706] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572386675worker1 # [ 147.833461] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6676worker1 # [ 147.857456] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:56Z level=info event=niks3_push_started pid=17816677worker1 # [ 148.091555] nix-grpc-daemon[1781]: time=2026-09-21T12:59:56.934Z level=INFO msg="Configuring client TLS" cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.pem key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/worker1.key ca=""6678worker1 # [ 148.290828] nix-daemon[849]: accepted connection from pid 1794, user nix-grpc-daemon (trusted)6679worker1 # [ 148.303221] nix-daemon[849]: reaped child process 1797, status = succeeded6680lb # [ 148.358417] envoy[922]: conn peer=[2001:db8:1::1]:56786 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9585 tx=4066 ms=63246681lb # [ 148.362268] envoy[922]: conn peer=[2001:db8:1::1]:44270 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2975 tx=1405 ms=110306682lb # [ 148.365932] envoy[922]: conn peer=[2001:db8:1::3]:34528 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=4007 tx=901 ms=432136683worker1 # [ 148.508381] nix-daemon[849]: accepted connection from pid 1799, user nix-grpc-daemon (trusted)6684worker1 # [ 148.522115] nix-daemon[849]: reaped child process 1802, status = succeeded6685lb # [ 148.541115] niks3-server[1174]: 2026/09/21 12:59:57 INFO Received uploads request method=POST path=/api/pending_closures6686worker1 # [ 148.552639] nix-grpc-daemon[1781]: time=2026-09-21T12:59:57.396Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6687worker1 # [ 148.555995] nix-grpc-daemon[1781]: time=2026-09-21T12:59:57.397Z level=INFO msg="Uploading cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv (504B)"6688lb # [ 148.572172] niks3-server[1174]: 2026/09/21 12:59:57 INFO Registered completed upload object_key=nar/1ifhv0ll3mnj6a2hrp33fzwnm2yl2zjxwqfrqjriibhs5cbnyiwi.nar.zst6689lb # [ 148.581828] niks3-server[1174]: 2026/09/21 12:59:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6690lb # [ 148.586618] niks3-server[1174]: 2026/09/21 12:59:57 INFO Signed narinfos id=52 count=16691worker1 # [ 148.592991] nix-grpc-daemon[1781]: time=2026-09-21T12:59:57.436Z level=INFO msg="Uploading 1 narinfos"6692lb # [ 148.601571] niks3-server[1174]: 2026/09/21 12:59:57 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.ls6693lb # [ 148.605489] niks3-server[1174]: 2026/09/21 12:59:57 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6694worker1 # [ 148.615299] nix-grpc-daemon[1781]: time=2026-09-21T12:59:57.458Z level=INFO msg="Upload complete. (519ms)"6695lb # [ 148.612181] niks3-server[1174]: 2026/09/21 12:59:57 INFO Completed upload id=526696worker1 # [ 148.617783] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=0 paths=1 nar_bytes_in=5046697worker1 # [ 148.621653] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386698lb # [ 148.618107] niks3-server[1174]: 2026/09/21 12:59:57 INFO Registered completed upload object_key=cv151lgjr4lv54sqzb49xk9w1ak9xsm9.narinfo6699worker1 # [ 148.629417] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6700worker1 # [ 148.632173] nix-daemon[849]: reaped child process 1779, status = succeeded6701worker1 # [ 148.637469] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6702worker1 # [ 149.461409] nix-daemon[849]: reaped child process 1806, status = succeeded6703worker1 # [ 149.647863] nix-daemon[849]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)6704worker1 # [ 149.660282] nix-daemon[849]: reaped child process 1825, status = succeeded6705lb # [ 149.677623] niks3-server[1174]: 2026/09/21 12:59:58 INFO Received uploads request method=POST path=/api/pending_closures6706worker1 # [ 149.689336] nix-grpc-daemon[1781]: time=2026-09-21T12:59:58.532Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6707worker1 # [ 149.690912] nix-grpc-daemon[1781]: time=2026-09-21T12:59:58.534Z level=INFO msg="Uploading wfziwgf0gxmjhy17lyfmn47628s166p7-farm-a (128B)"6708lb # [ 149.713794] niks3-server[1174]: 2026/09/21 12:59:58 INFO Registered completed upload object_key=log/cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv6709lb # [ 149.724680] niks3-server[1174]: 2026/09/21 12:59:58 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6710lb # [ 149.735218] niks3-server[1174]: 2026/09/21 12:59:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6711lb # [ 149.740202] niks3-server[1174]: 2026/09/21 12:59:58 INFO Signed narinfos id=53 count=16712worker1 # [ 149.744161] nix-grpc-daemon[1781]: time=2026-09-21T12:59:58.586Z level=INFO msg="Uploading 1 narinfos"6713lb # [ 149.744743] niks3-server[1174]: 2026/09/21 12:59:58 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.ls6714lb # [ 149.753211] niks3-server[1174]: 2026/09/21 12:59:58 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6715lb # [ 149.758968] niks3-server[1174]: 2026/09/21 12:59:58 INFO Completed upload id=536716worker1 # [ 149.762145] nix-grpc-daemon[1781]: time=2026-09-21T12:59:58.605Z level=INFO msg="Upload complete. (317ms)"6717lb # [ 149.761142] niks3-server[1174]: 2026/09/21 12:59:58 INFO Registered completed upload object_key=wfziwgf0gxmjhy17lyfmn47628s166p7.narinfo6718worker2 # [ 149.793978] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:58Z level=debug event=assigned drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv worker=192.168.1.3:50051 clients=16719worker1 # [ 149.767343] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=1 drv=cv151lgjr4lv54sqzb49xk9w1ak9xsm9-farm-a.drv assign_id=6 outputs=16720worker2 # [ 149.799331] nix-grpc-daemon[1488]: ts=2026-09-21T12:59:58Z level=debug event=want client=5 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv cached=06721worker1 # [ 149.775455] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386722worker1 # [ 149.785369] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572386723worker1 # [ 149.789052] nix-daemon[849]: reaped child process 1804, status = succeeded6724worker1 # [ 149.790857] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572386725worker1 # [ 149.795112] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6726worker1 # [ 149.983140] nix-daemon[849]: accepted connection from pid 1829, user nix-grpc-daemon (trusted)6727worker1 # [ 149.997268] nix-daemon[849]: reaped child process 1832, status = succeeded6728worker1 # [ 150.193361] nix-daemon[849]: accepted connection from pid 1834, user nix-grpc-daemon (trusted)6729worker1 # [ 150.206902] nix-daemon[849]: reaped child process 1837, status = succeeded6730lb # [ 150.222333] niks3-server[1174]: 2026/09/21 12:59:59 INFO Received uploads request method=POST path=/api/pending_closures6731worker1 # [ 150.234249] nix-grpc-daemon[1781]: time=2026-09-21T12:59:59.077Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6732worker1 # [ 150.235769] nix-grpc-daemon[1781]: time=2026-09-21T12:59:59.079Z level=INFO msg="Uploading fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv (504B)"6733lb # [ 150.255898] niks3-server[1174]: 2026/09/21 12:59:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6734lb # [ 150.260901] niks3-server[1174]: 2026/09/21 12:59:59 INFO Signed narinfos id=54 count=16735worker1 # [ 150.267267] nix-grpc-daemon[1781]: time=2026-09-21T12:59:59.110Z level=INFO msg="Uploading 1 narinfos"6736lb # [ 150.270145] niks3-server[1174]: 2026/09/21 12:59:59 INFO Registered completed upload object_key=nar/170l67nd7hvasq2l36jibfpivx7r9ssdy0wlzifggpplhcgf2rk0.nar.zst6737lb # [ 150.276158] niks3-server[1174]: 2026/09/21 12:59:59 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.ls6738lb # [ 150.286319] niks3-server[1174]: 2026/09/21 12:59:59 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6739worker1 # [ 150.295828] nix-grpc-daemon[1781]: time=2026-09-21T12:59:59.139Z level=INFO msg="Upload complete. (494ms)"6740lb # [ 150.292858] niks3-server[1174]: 2026/09/21 12:59:59 INFO Completed upload id=546741worker1 # [ 150.298135] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=0 paths=1 nar_bytes_in=5046742lb # [ 150.296480] niks3-server[1174]: 2026/09/21 12:59:59 INFO Registered completed upload object_key=fggw0ziag6jlrc830j6x1b6qw68kybhf.narinfo6743worker1 # [ 150.303686] nix-grpc-daemon[1721]: ts=2026-09-21T12:59:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386744worker1 # [ 150.309697] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6745worker1 # [ 150.312166] nix-daemon[849]: reaped child process 1827, status = succeeded6746worker1 # [ 150.317794] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6747worker1 # [ 151.142975] nix-daemon[849]: reaped child process 1841, status = succeeded6748worker1 # [ 151.322818] nix-daemon[849]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6749worker1 # [ 151.335589] nix-daemon[849]: reaped child process 1860, status = succeeded6750lb # [ 151.352348] niks3-server[1174]: 2026/09/21 13:00:00 INFO Received uploads request method=POST path=/api/pending_closures6751worker1 # [ 151.365630] nix-grpc-daemon[1781]: time=2026-09-21T13:00:00.208Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6752worker1 # [ 151.367312] nix-grpc-daemon[1781]: time=2026-09-21T13:00:00.210Z level=INFO msg="Uploading 63zyaryk3r1ikmmzi1w30vadf20kdw3y-farm-a (128B)"6753lb # [ 151.385622] niks3-server[1174]: 2026/09/21 13:00:00 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6754lb # [ 151.399533] niks3-server[1174]: 2026/09/21 13:00:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6755lb # [ 151.405115] niks3-server[1174]: 2026/09/21 13:00:00 INFO Signed narinfos id=55 count=16756worker1 # [ 151.411492] nix-grpc-daemon[1781]: time=2026-09-21T13:00:00.254Z level=INFO msg="Uploading 1 narinfos"6757lb # [ 151.411230] niks3-server[1174]: 2026/09/21 13:00:00 INFO Registered completed upload object_key=log/fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv6758lb # [ 151.419713] niks3-server[1174]: 2026/09/21 13:00:00 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.ls6759lb # [ 151.426334] niks3-server[1174]: 2026/09/21 13:00:00 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6760lb # [ 151.430719] niks3-server[1174]: 2026/09/21 13:00:00 INFO Completed upload id=556761worker1 # [ 151.434340] nix-grpc-daemon[1781]: time=2026-09-21T13:00:00.277Z level=INFO msg="Upload complete. (309ms)"6762lb # [ 151.433173] niks3-server[1174]: 2026/09/21 13:00:00 INFO Registered completed upload object_key=63zyaryk3r1ikmmzi1w30vadf20kdw3y.narinfo6763worker1 # [ 151.440593] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=1 drv=fggw0ziag6jlrc830j6x1b6qw68kybhf-farm-a.drv assign_id=7 outputs=16764worker2 # [ 151.466544] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:00Z level=debug event=assigned drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv worker=192.168.1.3:50051 clients=16765worker1 # [ 151.446549] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386766worker2 # [ 151.475854] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:00Z level=debug event=want client=5 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv cached=06767worker1 # [ 151.456370] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572386768worker1 # [ 151.461920] nix-daemon[849]: reaped child process 1839, status = succeeded6769worker1 # [ 151.463655] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572386770worker1 # [ 151.467658] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6771worker1 # [ 151.643230] nix-daemon[849]: accepted connection from pid 1864, user nix-grpc-daemon (trusted)6772worker1 # [ 151.658376] nix-daemon[849]: reaped child process 1867, status = succeeded6773worker1 # [ 151.841101] nix-daemon[849]: accepted connection from pid 1869, user nix-grpc-daemon (trusted)6774worker1 # [ 151.853440] nix-daemon[849]: reaped child process 1872, status = succeeded6775lb # [ 151.867696] niks3-server[1174]: 2026/09/21 13:00:00 INFO Received uploads request method=POST path=/api/pending_closures6776worker1 # [ 151.879160] nix-grpc-daemon[1781]: time=2026-09-21T13:00:00.722Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6777worker1 # [ 151.880730] nix-grpc-daemon[1781]: time=2026-09-21T13:00:00.724Z level=INFO msg="Uploading j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv (504B)"6778lb # [ 151.895907] niks3-server[1174]: 2026/09/21 13:00:00 INFO Registered completed upload object_key=nar/0qr7hdc916yrqpvqx2bwc5l0rwy3zm6rsdlcf8rk9iid53b80xsn.nar.zst6779lb # [ 151.906920] niks3-server[1174]: 2026/09/21 13:00:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6780lb # [ 151.911097] niks3-server[1174]: 2026/09/21 13:00:00 INFO Signed narinfos id=56 count=16781worker1 # [ 151.917251] nix-grpc-daemon[1781]: time=2026-09-21T13:00:00.760Z level=INFO msg="Uploading 1 narinfos"6782lb # [ 151.920356] niks3-server[1174]: 2026/09/21 13:00:00 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.ls6783lb # [ 151.929386] niks3-server[1174]: 2026/09/21 13:00:00 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6784lb # [ 151.934589] niks3-server[1174]: 2026/09/21 13:00:00 INFO Completed upload id=566785worker1 # [ 151.937506] nix-grpc-daemon[1781]: time=2026-09-21T13:00:00.780Z level=INFO msg="Upload complete. (463ms)"6786worker1 # [ 151.940997] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=0 paths=1 nar_bytes_in=5046787lb # [ 151.940072] niks3-server[1174]: 2026/09/21 13:00:00 INFO Registered completed upload object_key=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz.narinfo6788worker1 # [ 151.947560] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386789worker1 # [ 151.956903] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6790worker1 # [ 151.960587] nix-daemon[849]: reaped child process 1862, status = succeeded6791worker1 # [ 151.965639] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6792worker1 # [ 152.995726] nix-daemon[849]: reaped child process 1876, status = succeeded6793worker1 # [ 153.172992] nix-daemon[849]: accepted connection from pid 1892, user nix-grpc-daemon (trusted)6794worker1 # [ 153.187188] nix-daemon[849]: reaped child process 1895, status = succeeded6795lb # [ 153.201740] niks3-server[1174]: 2026/09/21 13:00:02 INFO Received uploads request method=POST path=/api/pending_closures6796worker1 # [ 153.213385] nix-grpc-daemon[1781]: time=2026-09-21T13:00:02.056Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6797worker1 # [ 153.214920] nix-grpc-daemon[1781]: time=2026-09-21T13:00:02.058Z level=INFO msg="Uploading pnim4ls33hwi0fl1wv96w4yj4afz9w4m-farm-a (128B)"6798lb # [ 153.229329] niks3-server[1174]: 2026/09/21 13:00:02 INFO Registered completed upload object_key=log/j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv6799lb # [ 153.249218] niks3-server[1174]: 2026/09/21 13:00:02 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6800lb # [ 153.258072] niks3-server[1174]: 2026/09/21 13:00:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6801worker1 # [ 153.265973] nix-grpc-daemon[1781]: time=2026-09-21T13:00:02.109Z level=INFO msg="Uploading 1 narinfos"6802lb # [ 153.262819] niks3-server[1174]: 2026/09/21 13:00:02 INFO Signed narinfos id=57 count=16803lb # [ 153.267792] niks3-server[1174]: 2026/09/21 13:00:02 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.ls6804lb # [ 153.275562] niks3-server[1174]: 2026/09/21 13:00:02 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6805lb # [ 153.281250] niks3-server[1174]: 2026/09/21 13:00:02 INFO Completed upload id=576806worker1 # [ 153.284689] nix-grpc-daemon[1781]: time=2026-09-21T13:00:02.128Z level=INFO msg="Upload complete. (311ms)"6807lb # [ 153.283617] niks3-server[1174]: 2026/09/21 13:00:02 INFO Registered completed upload object_key=pnim4ls33hwi0fl1wv96w4yj4afz9w4m.narinfo6808worker1 # [ 153.290478] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=1 drv=j7ssjgj9yiiadgfwc3qmhbma7i6ylnrz-farm-a.drv assign_id=8 outputs=16809worker2 # [ 153.316807] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:02Z level=debug event=assigned drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv worker=192.168.1.3:50051 clients=16810worker2 # [ 153.326910] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:02Z level=debug event=want client=5 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv cached=06811worker1 # [ 153.305596] nix-daemon[849]: reaped child process 1874, status = succeeded6812worker1 # [ 153.340154] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386813worker1 # [ 153.352684] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572386814worker1 # [ 153.365239] nix-grpc-daemon[1781]: time=2026-09-21T13:00:02.208Z level=INFO msg="All 1 paths already cached"6815worker1 # [ 153.372349] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572386816worker1 # [ 153.382364] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6817worker1 # [ 153.573345] nix-daemon[849]: accepted connection from pid 1899, user nix-grpc-daemon (trusted)6818worker1 # [ 153.587663] nix-daemon[849]: reaped child process 1902, status = succeeded6819worker1 # [ 153.762513] nix-daemon[849]: accepted connection from pid 1904, user nix-grpc-daemon (trusted)6820worker1 # [ 153.776697] nix-daemon[849]: reaped child process 1907, status = succeeded6821lb # [ 153.790486] niks3-server[1174]: 2026/09/21 13:00:02 INFO Received uploads request method=POST path=/api/pending_closures6822worker1 # [ 153.801638] nix-grpc-daemon[1781]: time=2026-09-21T13:00:02.645Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6823worker1 # [ 153.803149] nix-grpc-daemon[1781]: time=2026-09-21T13:00:02.646Z level=INFO msg="Uploading 6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv (624B)"6824lb # [ 153.818314] niks3-server[1174]: 2026/09/21 13:00:02 INFO Registered completed upload object_key=nar/06n0qn8haxwg1vfa3qwa7h14mzz5fz7nizpv8bn995f4gddci6y6.nar.zst6825lb # [ 153.829768] niks3-server[1174]: 2026/09/21 13:00:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6826worker1 # [ 153.837821] nix-grpc-daemon[1781]: time=2026-09-21T13:00:02.681Z level=INFO msg="Uploading 1 narinfos"6827lb # [ 153.834895] niks3-server[1174]: 2026/09/21 13:00:02 INFO Signed narinfos id=58 count=16828lb # [ 153.840687] niks3-server[1174]: 2026/09/21 13:00:02 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.ls6829lb # [ 153.851833] niks3-server[1174]: 2026/09/21 13:00:02 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6830worker1 # [ 153.860957] nix-grpc-daemon[1781]: time=2026-09-21T13:00:02.704Z level=INFO msg="Upload complete. (466ms)"6831lb # [ 153.857716] niks3-server[1174]: 2026/09/21 13:00:02 INFO Completed upload id=586832worker1 # [ 153.863212] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=0 paths=1 nar_bytes_in=6246833lb # [ 153.860736] niks3-server[1174]: 2026/09/21 13:00:02 INFO Registered completed upload object_key=6cjjd3vbzygfms2ccxvnmabbgq5pv310.narinfo6834worker1 # [ 153.868059] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386835worker1 # [ 153.874850] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6836worker1 # [ 153.877295] nix-daemon[849]: reaped child process 1897, status = succeeded6837worker1 # [ 153.883139] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6838worker1 # [ 155.100893] nix-daemon[849]: reaped child process 1911, status = succeeded6839worker1 # [ 155.275704] nix-daemon[849]: accepted connection from pid 1927, user nix-grpc-daemon (trusted)6840worker1 # [ 155.288260] nix-daemon[849]: reaped child process 1930, status = succeeded6841lb # [ 155.304664] niks3-server[1174]: 2026/09/21 13:00:04 INFO Received uploads request method=POST path=/api/pending_closures6842worker1 # [ 155.318158] nix-grpc-daemon[1781]: time=2026-09-21T13:00:04.161Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6843worker1 # [ 155.319745] nix-grpc-daemon[1781]: time=2026-09-21T13:00:04.163Z level=INFO msg="Uploading 7gfg490kigdqainqjk27vmks1h3wpv5q-farm-b (176B)"6844lb # [ 155.348690] niks3-server[1174]: 2026/09/21 13:00:04 INFO Registered completed upload object_key=log/6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv6845lb # [ 155.359214] niks3-server[1174]: 2026/09/21 13:00:04 INFO Registered completed upload object_key=nar/07q9c11jk9djv06znvadbhgp1j3kwqf4z0z60kwx2sjqjkpssnf0.nar.zst6846lb # [ 155.366393] niks3-server[1174]: 2026/09/21 13:00:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6847worker1 # [ 155.375447] nix-grpc-daemon[1781]: time=2026-09-21T13:00:04.218Z level=INFO msg="Uploading 1 narinfos"6848lb # [ 155.371726] niks3-server[1174]: 2026/09/21 13:00:04 INFO Signed narinfos id=59 count=16849lb # [ 155.377718] niks3-server[1174]: 2026/09/21 13:00:04 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.ls6850lb # [ 155.386736] niks3-server[1174]: 2026/09/21 13:00:04 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6851lb # [ 155.392185] niks3-server[1174]: 2026/09/21 13:00:04 INFO Completed upload id=596852worker1 # [ 155.395660] nix-grpc-daemon[1781]: time=2026-09-21T13:00:04.239Z level=INFO msg="Upload complete. (303ms)"6853lb # [ 155.393927] niks3-server[1174]: 2026/09/21 13:00:04 INFO Registered completed upload object_key=7gfg490kigdqainqjk27vmks1h3wpv5q.narinfo6854worker1 # [ 155.401540] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=1 drv=6cjjd3vbzygfms2ccxvnmabbgq5pv310-farm-b.drv assign_id=9 outputs=16855worker2 # [ 155.428824] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:04Z level=debug event=assigned drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv worker=192.168.1.3:50051 clients=16856worker2 # [ 155.438896] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:04Z level=debug event=want client=5 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv cached=06857worker1 # [ 155.416676] nix-daemon[849]: reaped child process 1909, status = succeeded6858worker1 # [ 155.446418] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386859worker1 # [ 155.459267] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572386860worker1 # [ 155.472123] nix-grpc-daemon[1781]: time=2026-09-21T13:00:04.315Z level=INFO msg="All 1 paths already cached"6861worker1 # [ 155.479781] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572386862worker1 # [ 155.487565] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6863worker1 # [ 155.684182] nix-daemon[849]: accepted connection from pid 1934, user nix-grpc-daemon (trusted)6864worker1 # [ 155.698354] nix-daemon[849]: reaped child process 1937, status = succeeded6865worker1 # [ 155.890514] nix-daemon[849]: accepted connection from pid 1939, user nix-grpc-daemon (trusted)6866worker1 # [ 155.904437] nix-daemon[849]: reaped child process 1942, status = succeeded6867lb # [ 155.918970] niks3-server[1174]: 2026/09/21 13:00:04 INFO Received uploads request method=POST path=/api/pending_closures6868worker1 # [ 155.930105] nix-grpc-daemon[1781]: time=2026-09-21T13:00:04.773Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6869worker1 # [ 155.931628] nix-grpc-daemon[1781]: time=2026-09-21T13:00:04.775Z level=INFO msg="Uploading 6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv (624B)"6870lb # [ 155.950611] niks3-server[1174]: 2026/09/21 13:00:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6871lb # [ 155.955363] niks3-server[1174]: 2026/09/21 13:00:04 INFO Registered completed upload object_key=nar/0w4dmpczwycj1p7ngvcwdynpk5hh6r5ivyz44k3lx2gh5vpk4l0i.nar.zst6872worker1 # [ 155.965873] nix-grpc-daemon[1781]: time=2026-09-21T13:00:04.809Z level=INFO msg="Uploading 1 narinfos"6873lb # [ 155.962587] niks3-server[1174]: 2026/09/21 13:00:04 INFO Signed narinfos id=60 count=16874lb # [ 155.970391] niks3-server[1174]: 2026/09/21 13:00:04 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.ls6875lb # [ 155.981475] niks3-server[1174]: 2026/09/21 13:00:04 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6876worker1 # [ 155.993620] nix-grpc-daemon[1781]: time=2026-09-21T13:00:04.837Z level=INFO msg="Upload complete. (492ms)"6877lb # [ 155.990428] niks3-server[1174]: 2026/09/21 13:00:04 INFO Completed upload id=606878lb # [ 155.992837] niks3-server[1174]: 2026/09/21 13:00:04 INFO Registered completed upload object_key=6cs4w0lyx4kldp39y5k29p7c0234cffw.narinfo6879worker1 # [ 155.996583] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=0 paths=1 nar_bytes_in=6246880worker1 # [ 156.002566] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386881worker1 # [ 156.013500] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6882worker1 # [ 156.017831] nix-daemon[849]: reaped child process 1932, status = succeeded6883worker1 # [ 156.024140] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6884worker1 # [ 157.238035] nix-daemon[849]: reaped child process 1946, status = succeeded6885worker1 # [ 157.413410] nix-daemon[849]: accepted connection from pid 1962, user nix-grpc-daemon (trusted)6886worker1 # [ 157.427418] nix-daemon[849]: reaped child process 1965, status = succeeded6887lb # [ 157.443254] niks3-server[1174]: 2026/09/21 13:00:06 INFO Received uploads request method=POST path=/api/pending_closures6888worker1 # [ 157.457350] nix-grpc-daemon[1781]: time=2026-09-21T13:00:06.298Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6889worker1 # [ 157.458734] nix-grpc-daemon[1781]: time=2026-09-21T13:00:06.298Z level=INFO msg="Uploading xxjh88rg0bhdcm1c6b12clv7l5hfbaa9-farm-b (176B)"6890lb # [ 157.484891] niks3-server[1174]: 2026/09/21 13:00:06 INFO Registered completed upload object_key=nar/1a8ws2jbn6nqpggi64yfgxvlwny3nffvyw6svm59is55hih5ppq4.nar.zst6891lb # [ 157.494248] niks3-server[1174]: 2026/09/21 13:00:06 INFO Registered completed upload object_key=log/6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv6892lb # [ 157.501828] niks3-server[1174]: 2026/09/21 13:00:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6893lb # [ 157.506690] niks3-server[1174]: 2026/09/21 13:00:06 INFO Signed narinfos id=61 count=16894worker1 # [ 157.512476] nix-grpc-daemon[1781]: time=2026-09-21T13:00:06.355Z level=INFO msg="Uploading 1 narinfos"6895lb # [ 157.513364] niks3-server[1174]: 2026/09/21 13:00:06 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.ls6896lb # [ 157.522325] niks3-server[1174]: 2026/09/21 13:00:06 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6897lb # [ 157.526901] niks3-server[1174]: 2026/09/21 13:00:06 INFO Completed upload id=616898worker1 # [ 157.530503] nix-grpc-daemon[1781]: time=2026-09-21T13:00:06.373Z level=INFO msg="Upload complete. (298ms)"6899lb # [ 157.529707] niks3-server[1174]: 2026/09/21 13:00:06 INFO Registered completed upload object_key=xxjh88rg0bhdcm1c6b12clv7l5hfbaa9.narinfo6900worker2 # [ 157.565867] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:06Z level=debug event=assigned drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv worker=192.168.1.3:50051 clients=16901worker1 # [ 157.536597] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=1 drv=6cs4w0lyx4kldp39y5k29p7c0234cffw-farm-b.drv assign_id=10 outputs=16902worker2 # [ 157.572622] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:06Z level=debug event=want client=5 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv cached=06903worker1 # [ 157.551365] nix-daemon[849]: reaped child process 1944, status = succeeded6904worker1 # [ 157.583186] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386905worker1 # [ 157.595645] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572386906worker1 # [ 157.608685] nix-grpc-daemon[1781]: time=2026-09-21T13:00:06.452Z level=INFO msg="All 1 paths already cached"6907worker1 # [ 157.615740] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572386908worker1 # [ 157.624354] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6909worker1 # [ 157.810957] nix-daemon[849]: accepted connection from pid 1969, user nix-grpc-daemon (trusted)6910worker1 # [ 157.826134] nix-daemon[849]: reaped child process 1972, status = succeeded6911worker1 # [ 158.005485] nix-daemon[849]: accepted connection from pid 1974, user nix-grpc-daemon (trusted)6912worker1 # [ 158.020861] nix-daemon[849]: reaped child process 1977, status = succeeded6913lb # [ 158.033377] niks3-server[1174]: 2026/09/21 13:00:06 INFO Received uploads request method=POST path=/api/pending_closures6914worker1 # [ 158.047288] nix-grpc-daemon[1781]: time=2026-09-21T13:00:06.890Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6915worker1 # [ 158.048819] nix-grpc-daemon[1781]: time=2026-09-21T13:00:06.892Z level=INFO msg="Uploading 8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv (624B)"6916lb # [ 158.066443] niks3-server[1174]: 2026/09/21 13:00:06 INFO Registered completed upload object_key=nar/1dh50s9lbv3zd84hw5q9axrhfi7ms38cl3gyrb9gas00gni4siz2.nar.zst6917lb # [ 158.076799] niks3-server[1174]: 2026/09/21 13:00:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6918worker1 # [ 158.084479] nix-grpc-daemon[1781]: time=2026-09-21T13:00:06.927Z level=INFO msg="Uploading 1 narinfos"6919lb # [ 158.081711] niks3-server[1174]: 2026/09/21 13:00:06 INFO Signed narinfos id=62 count=16920lb # [ 158.089384] niks3-server[1174]: 2026/09/21 13:00:07 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.ls6921lb # [ 158.098544] niks3-server[1174]: 2026/09/21 13:00:07 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6922lb # [ 158.106416] niks3-server[1174]: 2026/09/21 13:00:07 INFO Completed upload id=626923worker1 # [ 158.109262] nix-grpc-daemon[1781]: time=2026-09-21T13:00:06.952Z level=INFO msg="Upload complete. (472ms)"6924lb # [ 158.109947] niks3-server[1174]: 2026/09/21 13:00:07 INFO Registered completed upload object_key=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz.narinfo6925worker1 # [ 158.112738] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=0 paths=1 nar_bytes_in=6246926worker1 # [ 158.118294] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386927worker1 # [ 158.126287] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6928worker1 # [ 158.132652] nix-daemon[849]: reaped child process 1967, status = succeeded6929worker1 # [ 158.138218] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6930worker1 # [ 159.567319] nix-daemon[849]: reaped child process 1981, status = succeeded6931worker1 # [ 159.749767] nix-daemon[849]: accepted connection from pid 1997, user nix-grpc-daemon (trusted)6932worker1 # [ 159.763833] nix-daemon[849]: reaped child process 2000, status = succeeded6933lb # [ 159.780121] niks3-server[1174]: 2026/09/21 13:00:08 INFO Received uploads request method=POST path=/api/pending_closures6934worker1 # [ 159.789719] nix-grpc-daemon[1781]: time=2026-09-21T13:00:08.632Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6935worker1 # [ 159.791260] nix-grpc-daemon[1781]: time=2026-09-21T13:00:08.634Z level=INFO msg="Uploading wha0h9jdcpcy0srjm92hg7pyknwdqp6h-farm-b (176B)"6936lb # [ 159.814246] niks3-server[1174]: 2026/09/21 13:00:08 INFO Registered completed upload object_key=nar/1xpjwixqgm25ifjqv3ldg9cbq2njxf26272sngvq3zhy6zr1mvwg.nar.zst6937lb # [ 159.824244] niks3-server[1174]: 2026/09/21 13:00:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6938lb # [ 159.829112] niks3-server[1174]: 2026/09/21 13:00:08 INFO Signed narinfos id=63 count=16939worker1 # [ 159.835359] nix-grpc-daemon[1781]: time=2026-09-21T13:00:08.678Z level=INFO msg="Uploading 1 narinfos"6940lb # [ 159.836210] niks3-server[1174]: 2026/09/21 13:00:08 INFO Registered completed upload object_key=log/8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv6941lb # [ 159.844684] niks3-server[1174]: 2026/09/21 13:00:08 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.ls6942lb # [ 159.852789] niks3-server[1174]: 2026/09/21 13:00:08 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6943lb # [ 159.858363] niks3-server[1174]: 2026/09/21 13:00:08 INFO Completed upload id=636944lb # [ 159.860565] niks3-server[1174]: 2026/09/21 13:00:08 INFO Registered completed upload object_key=wha0h9jdcpcy0srjm92hg7pyknwdqp6h.narinfo6945worker1 # [ 159.863458] nix-grpc-daemon[1781]: time=2026-09-21T13:00:08.706Z level=INFO msg="Upload complete. (316ms)"6946worker2 # [ 159.897664] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:08Z level=debug event=assigned drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv worker=192.168.1.3:50051 clients=16947worker1 # [ 159.872098] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=1 drv=8pwrbica5d7v9bivkz2fiy3z2fjlj9dz-farm-b.drv assign_id=11 outputs=16948worker1 # [ 159.878447] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386949worker2 # [ 159.904880] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:08Z level=debug event=want client=5 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv cached=06950worker1 # [ 159.890576] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572386951worker1 # [ 159.898062] nix-daemon[849]: reaped child process 1979, status = succeeded6952worker1 # [ 159.900577] nix-grpc-daemon[1781]: time=2026-09-21T13:00:08.744Z level=INFO msg="All 2 paths already cached"6953worker1 # [ 159.905291] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572386954worker1 # [ 159.909370] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6955worker1 # [ 160.087448] nix-daemon[849]: accepted connection from pid 2004, user nix-grpc-daemon (trusted)6956worker1 # [ 160.101543] nix-daemon[849]: reaped child process 2007, status = succeeded6957worker1 # [ 160.282801] nix-daemon[849]: accepted connection from pid 2009, user nix-grpc-daemon (trusted)6958worker1 # [ 160.296933] nix-daemon[849]: reaped child process 2012, status = succeeded6959lb # [ 160.311573] niks3-server[1174]: 2026/09/21 13:00:09 INFO Received uploads request method=POST path=/api/pending_closures6960worker1 # [ 160.322595] nix-grpc-daemon[1781]: time=2026-09-21T13:00:09.166Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6961worker1 # [ 160.324474] nix-grpc-daemon[1781]: time=2026-09-21T13:00:09.167Z level=INFO msg="Uploading 2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv (752B)"6962lb # [ 160.340597] niks3-server[1174]: 2026/09/21 13:00:09 INFO Registered completed upload object_key=nar/10gk7wwjbmc835pm215qvrdyjshy6nblydlws97ya3kcny5wz30c.nar.zst6963lb # [ 160.352089] niks3-server[1174]: 2026/09/21 13:00:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6964worker1 # [ 160.360720] nix-grpc-daemon[1781]: time=2026-09-21T13:00:09.204Z level=INFO msg="Uploading 1 narinfos"6965lb # [ 160.357194] niks3-server[1174]: 2026/09/21 13:00:09 INFO Signed narinfos id=64 count=16966lb # [ 160.363274] niks3-server[1174]: 2026/09/21 13:00:09 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.ls6967lb # [ 160.374268] niks3-server[1174]: 2026/09/21 13:00:09 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6968lb # [ 160.379494] niks3-server[1174]: 2026/09/21 13:00:09 INFO Completed upload id=646969worker1 # [ 160.385184] nix-grpc-daemon[1781]: time=2026-09-21T13:00:09.228Z level=INFO msg="Upload complete. (469ms)"6970worker1 # [ 160.386784] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=0 paths=1 nar_bytes_in=7526971lb # [ 160.382862] niks3-server[1174]: 2026/09/21 13:00:09 INFO Registered completed upload object_key=2gb1j4yk6j00fr2sscj3nqixv67k9b9g.narinfo6972worker1 # [ 160.392721] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386973worker1 # [ 160.402265] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6974worker1 # [ 160.404782] nix-daemon[849]: reaped child process 2002, status = succeeded6975worker1 # [ 160.410546] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)6976worker1 # [ 161.032777] nix-daemon[849]: reaped child process 2016, status = succeeded6977worker1 # [ 161.207094] nix-daemon[849]: accepted connection from pid 2032, user nix-grpc-daemon (trusted)6978worker1 # [ 161.218364] nix-daemon[849]: reaped child process 2035, status = succeeded6979lb # [ 161.234399] niks3-server[1174]: 2026/09/21 13:00:10 INFO Received uploads request method=POST path=/api/pending_closures6980worker1 # [ 161.246030] nix-grpc-daemon[1781]: time=2026-09-21T13:00:10.089Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6981worker1 # [ 161.247578] nix-grpc-daemon[1781]: time=2026-09-21T13:00:10.091Z level=INFO msg="Uploading 78nhq5a8myvxam7sxfszi5ihz2rlq2gs-farm-top (232B)"6982lb # [ 161.279178] niks3-server[1174]: 2026/09/21 13:00:10 INFO Registered completed upload object_key=nar/1gsz81ckdr6lfkhhqmxw3bg6xqqnlz2wx6dzhpycascv9pq13fg7.nar.zst6983lb # [ 161.286376] niks3-server[1174]: 2026/09/21 13:00:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6984worker1 # [ 161.294275] nix-grpc-daemon[1781]: time=2026-09-21T13:00:10.137Z level=INFO msg="Uploading 1 narinfos"6985lb # [ 161.290859] niks3-server[1174]: 2026/09/21 13:00:10 INFO Signed narinfos id=65 count=16986lb # [ 161.294235] niks3-server[1174]: 2026/09/21 13:00:10 INFO Registered completed upload object_key=log/2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv6987lb # [ 161.298640] niks3-server[1174]: 2026/09/21 13:00:10 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.ls6988lb # [ 161.310659] niks3-server[1174]: 2026/09/21 13:00:10 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6989lb # [ 161.313305] niks3-server[1174]: 2026/09/21 13:00:10 INFO Completed upload id=656990lb # [ 161.315124] niks3-server[1174]: 2026/09/21 13:00:10 INFO Registered completed upload object_key=78nhq5a8myvxam7sxfszi5ihz2rlq2gs.narinfo6991worker1 # [ 161.317926] nix-grpc-daemon[1781]: time=2026-09-21T13:00:10.161Z level=INFO msg="Upload complete. (300ms)"6992worker2 # [ 161.350477] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:10Z level=debug event=assigned drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv worker=192.168.1.3:50051 clients=16993worker1 # [ 161.325461] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=0 drv=2gb1j4yk6j00fr2sscj3nqixv67k9b9g-farm-top.drv assign_id=12 outputs=16994worker1 # [ 161.337523] nix-daemon[849]: reaped child process 2014, status = succeeded6995worker1 # [ 161.367874] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572386996worker1 # [ 161.380366] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572386997worker1 # [ 161.392943] nix-grpc-daemon[1781]: time=2026-09-21T13:00:10.236Z level=INFO msg="All 2 paths already cached"6998worker1 # [ 161.400219] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572386999worker1 # [ 161.408478] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)7000worker1 # [ 161.603235] nix-daemon[849]: accepted connection from pid 2039, user nix-grpc-daemon (trusted)7001worker1 # [ 161.618672] nix-daemon[849]: reaped child process 2042, status = succeeded7002worker1 # [ 161.800645] nix-daemon[849]: accepted connection from pid 2044, user nix-grpc-daemon (trusted)7003worker1 # [ 161.814419] nix-daemon[849]: reaped child process 2047, status = succeeded7004lb # [ 161.828154] niks3-server[1174]: 2026/09/21 13:00:10 INFO Received uploads request method=POST path=/api/pending_closures7005worker1 # [ 161.838981] nix-grpc-daemon[1781]: time=2026-09-21T13:00:10.682Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7006worker1 # [ 161.840946] nix-grpc-daemon[1781]: time=2026-09-21T13:00:10.683Z level=INFO msg="Uploading yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv (752B)"7007lb # [ 161.854600] niks3-server[1174]: 2026/09/21 13:00:10 INFO Registered completed upload object_key=nar/1ibq19fcqq0l9xim9ly5lzg1pk7qh90qsbpya453vnh8p619zvbs.nar.zst7008lb # [ 161.865158] niks3-server[1174]: 2026/09/21 13:00:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign7009worker1 # [ 161.873251] nix-grpc-daemon[1781]: time=2026-09-21T13:00:10.716Z level=INFO msg="Uploading 1 narinfos"7010lb # [ 161.870248] niks3-server[1174]: 2026/09/21 13:00:10 INFO Signed narinfos id=66 count=17011lb # [ 161.876647] niks3-server[1174]: 2026/09/21 13:00:10 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.ls7012lb # [ 161.887786] niks3-server[1174]: 2026/09/21 13:00:10 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete7013worker1 # [ 161.897289] nix-grpc-daemon[1781]: time=2026-09-21T13:00:10.740Z level=INFO msg="Upload complete. (472ms)"7014lb # [ 161.894150] niks3-server[1174]: 2026/09/21 13:00:10 INFO Completed upload id=667015lb # [ 161.897096] niks3-server[1174]: 2026/09/21 13:00:10 INFO Registered completed upload object_key=yjsciba0gpd99p30v1win6wnqyd4v7b6.narinfo7016worker1 # [ 161.899787] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=0 paths=1 nar_bytes_in=7527017worker1 # [ 161.905112] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572387018worker1 # [ 161.915793] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)7019worker1 # [ 161.919142] nix-daemon[849]: reaped child process 2037, status = succeeded7020worker1 # [ 161.924440] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)7021worker1 # [ 162.938360] nix-daemon[849]: reaped child process 2051, status = succeeded7022worker1 # [ 163.093503] nix-daemon[849]: accepted connection from pid 2067, user nix-grpc-daemon (trusted)7023worker1 # [ 163.108129] nix-daemon[849]: reaped child process 2070, status = succeeded7024lb # [ 163.123847] niks3-server[1174]: 2026/09/21 13:00:12 INFO Received uploads request method=POST path=/api/pending_closures7025worker1 # [ 163.136076] nix-grpc-daemon[1781]: time=2026-09-21T13:00:11.979Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7026worker1 # [ 163.137637] nix-grpc-daemon[1781]: time=2026-09-21T13:00:11.981Z level=INFO msg="Uploading qc1853riggbxq0nbn4qjdymnhx5pncg8-farm-top (232B)"7027lb # [ 163.156336] niks3-server[1174]: 2026/09/21 13:00:12 INFO Registered completed upload object_key=log/yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv7028lb # [ 163.167634] niks3-server[1174]: 2026/09/21 13:00:12 INFO Registered completed upload object_key=nar/1k6m6033nbrw39zmvqlrbvmg39ygyn710za2hl2fkpsxfx40hl13.nar.zst7029lb # [ 163.180246] niks3-server[1174]: 2026/09/21 13:00:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign7030worker1 # [ 163.188703] nix-grpc-daemon[1781]: time=2026-09-21T13:00:12.032Z level=INFO msg="Uploading 1 narinfos"7031lb # [ 163.185261] niks3-server[1174]: 2026/09/21 13:00:12 INFO Signed narinfos id=67 count=17032lb # [ 163.190222] niks3-server[1174]: 2026/09/21 13:00:12 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.ls7033lb # [ 163.199887] niks3-server[1174]: 2026/09/21 13:00:12 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete7034lb # [ 163.205404] niks3-server[1174]: 2026/09/21 13:00:12 INFO Completed upload id=677035worker1 # [ 163.208838] nix-grpc-daemon[1781]: time=2026-09-21T13:00:12.052Z level=INFO msg="Upload complete. (276ms)"7036lb # [ 163.207184] niks3-server[1174]: 2026/09/21 13:00:12 INFO Registered completed upload object_key=qc1853riggbxq0nbn4qjdymnhx5pncg8.narinfo7037worker2 # [ 163.241656] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:12Z level=debug event=assigned drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv worker=192.168.1.3:50051 clients=17038worker1 # [ 163.214681] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=1 drv=yjsciba0gpd99p30v1win6wnqyd4v7b6-farm-top.drv assign_id=13 outputs=17039worker1 # [ 163.229692] nix-daemon[849]: reaped child process 2049, status = succeeded7040worker1 # [ 163.259471] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572387041worker1 # [ 163.272128] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:572387042worker1 # [ 163.284716] nix-grpc-daemon[1781]: time=2026-09-21T13:00:12.128Z level=INFO msg="All 2 paths already cached"7043worker1 # [ 163.291871] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:572387044worker1 # [ 163.300400] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)7045worker1 # [ 163.478063] nix-daemon[849]: accepted connection from pid 2074, user nix-grpc-daemon (trusted)7046worker1 # [ 163.491791] nix-daemon[849]: reaped child process 2077, status = succeeded7047worker1 # [ 163.666101] nix-daemon[849]: accepted connection from pid 2079, user nix-grpc-daemon (trusted)7048worker1 # [ 163.678398] nix-daemon[849]: reaped child process 2082, status = succeeded7049lb # [ 163.692260] niks3-server[1174]: 2026/09/21 13:00:12 INFO Received uploads request method=POST path=/api/pending_closures7050worker1 # [ 163.703229] nix-grpc-daemon[1781]: time=2026-09-21T13:00:12.546Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7051worker1 # [ 163.704778] nix-grpc-daemon[1781]: time=2026-09-21T13:00:12.548Z level=INFO msg="Uploading yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv (752B)"7052lb # [ 163.721684] niks3-server[1174]: 2026/09/21 13:00:12 INFO Registered completed upload object_key=nar/1msrgzhylalg20dw693shbnzf48k8pvzp85llwgrnc970rnrliy7.nar.zst7053lb # [ 163.730633] niks3-server[1174]: 2026/09/21 13:00:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign7054lb # [ 163.736114] niks3-server[1174]: 2026/09/21 13:00:12 INFO Signed narinfos id=68 count=17055worker1 # [ 163.742457] nix-grpc-daemon[1781]: time=2026-09-21T13:00:12.585Z level=INFO msg="Uploading 1 narinfos"7056lb # [ 163.742604] niks3-server[1174]: 2026/09/21 13:00:12 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.ls7057lb # [ 163.755280] niks3-server[1174]: 2026/09/21 13:00:12 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete7058lb # [ 163.761302] niks3-server[1174]: 2026/09/21 13:00:12 INFO Completed upload id=687059worker1 # [ 163.766849] nix-grpc-daemon[1781]: time=2026-09-21T13:00:12.610Z level=INFO msg="Upload complete. (452ms)"7060lb # [ 163.763758] niks3-server[1174]: 2026/09/21 13:00:12 INFO Registered completed upload object_key=yym31x3l626nfpm249qapx1iw7y15mqb.narinfo7061worker1 # [ 163.769916] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=0 paths=1 nar_bytes_in=7527062worker1 # [ 163.775274] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:572387063worker1 # [ 163.783489] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)7064worker1 # [ 163.791655] nix-daemon[849]: reaped child process 2072, status = succeeded7065worker1 # [ 163.797062] nix-daemon[849]: accepted connection from pid 1721, user nix-grpc-daemon (trusted)7066worker1 # [ 164.423172] nix-daemon[849]: reaped child process 2086, status = succeeded7067worker1 # [ 164.601160] nix-daemon[849]: accepted connection from pid 2102, user nix-grpc-daemon (trusted)7068worker1 # [ 164.615247] nix-daemon[849]: reaped child process 2105, status = succeeded7069lb # [ 164.631182] niks3-server[1174]: 2026/09/21 13:00:13 INFO Received uploads request method=POST path=/api/pending_closures7070worker1 # [ 164.643407] nix-grpc-daemon[1781]: time=2026-09-21T13:00:13.486Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7071worker1 # [ 164.644906] nix-grpc-daemon[1781]: time=2026-09-21T13:00:13.488Z level=INFO msg="Uploading 0y5v4dgfslb2x47r2h6j0zjvpzg6frg6-farm-top (232B)"7072lb # [ 164.667314] niks3-server[1174]: 2026/09/21 13:00:13 INFO Registered completed upload object_key=log/yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv7073lb # [ 164.678803] niks3-server[1174]: 2026/09/21 13:00:13 INFO Registered completed upload object_key=nar/1h7kcfflb8hb0ylxshiprrirnrs580cwzkz50cz6il8jag61wy50.nar.zst7074lb # [ 164.686704] niks3-server[1174]: 2026/09/21 13:00:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign7075lb # [ 164.692061] niks3-server[1174]: 2026/09/21 13:00:13 INFO Signed narinfos id=69 count=17076worker1 # [ 164.698250] nix-grpc-daemon[1781]: time=2026-09-21T13:00:13.541Z level=INFO msg="Uploading 1 narinfos"7077lb # [ 164.697962] niks3-server[1174]: 2026/09/21 13:00:13 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.ls7078lb # [ 164.706517] niks3-server[1174]: 2026/09/21 13:00:13 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete7079lb # [ 164.712406] niks3-server[1174]: 2026/09/21 13:00:13 INFO Completed upload id=697080worker1 # [ 164.715402] nix-grpc-daemon[1781]: time=2026-09-21T13:00:13.558Z level=INFO msg="Upload complete. (311ms)"7081lb # [ 164.715609] niks3-server[1174]: 2026/09/21 13:00:13 INFO Registered completed upload object_key=0y5v4dgfslb2x47r2h6j0zjvpzg6frg6.narinfo7082worker1 # [ 164.722391] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57238 duration_s=0 drv=yym31x3l626nfpm249qapx1iw7y15mqb-farm-top.drv assign_id=14 outputs=17083worker1 # [ 164.734982] nix-daemon[849]: reaped child process 2084, status = succeeded7084client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 18.18 seconds)7085worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7086worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7087client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]'7088client # [ 164.919273] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7089client # Running as unit: gpu.service; invocation ID: 216d29a15a794e7d8026f7477a95a6db7090client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.07 seconds)7091lb: must succeed: curl -sf localhost:9901/clusters7092lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7093lb: must succeed: curl -sf localhost:9901/clusters7094lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7095worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7096worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)7097client # [ 165.154341] nix[1374]: don't know how to build these paths:7098client # [ 165.155262] nix[1374]: /nix/store/ynb3m2rijlmsagnc1dpj672gknpqrkcv-farm-top.drv7099worker1 # [ 165.139235] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:572387100worker2 # [ 165.168850] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436747101worker2 # [ 165.183338] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:14Z level=debug event=want client=6 drv=z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv cached=07102client # [ 165.171664] nix[1374]: warning: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}, waiting up to 30s for a worker7103lb: must succeed: curl -sf localhost:9901/clusters7104lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)7105lb: must succeed: curl -sf localhost:9901/clusters7106lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)7107worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7108worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)7109client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7110lb # [ 168.359693] envoy[922]: conn peer=[2001:db8:1::1]:46186 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27128 tx=11447 ms=178867111client # [ 195.174404] nix[1374]: z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv: no connected, non-draining worker for system 'x86_64-linux' offers features {gpu}7112client # [ 195.181169] nix[1374]: error: dependency 'z5cr5v9sb2xcx4fmgc48vvwyl2m8rzzs-farm-a.drv' failed7113client # [ 195.212978] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7114client # [ 195.216134] systemd[1]: gpu.service: Failed with result 'exit-code'.7115client # [ 195.219658] systemd[1]: gpu.service: Consumed 145ms CPU time over 30.299s wall clock time, 37.4M memory peak, 5K incoming IP traffic, 4.9K outgoing IP traffic.7116client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.82 seconds)7117client: must succeed: journalctl -u gpu -o cat7118client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)7119lb: must succeed: curl -sf localhost:9901/clusters7120lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)7121lb: must succeed: curl -sf localhost:9901/clusters7122lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7123worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics7124worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7125(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 49.71 seconds)7126subtest: low disk drains a worker and builds go to the other7127worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7128worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 0.94 seconds)7129worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7130worker1 # [ 197.341338] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:46Z level=info event=unhealthy reason=min_free7131worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.10 seconds)7132client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&27133lb # [ 198.362227] envoy[922]: conn peer=[2001:db8:1::1]:44590 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=895 tx=497 ms=300357134client # don't know how to build these paths:7135client # /nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7136worker2 # [ 198.626132] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:47Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:436747137worker2 # [ 198.631836] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436747138worker2 # [ 198.651937] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:47Z level=debug event=want client=7 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv cached=07139worker2 # [ 198.657163] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:47Z level=debug event=assigned drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv worker=192.168.1.4:50051 clients=17140worker2 # [ 198.701370] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436747141worker2 # [ 198.713724] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436747142client # copying 1 paths to 'lb:50051'...7143client # copying path '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' to 'lb:50051'...7144worker2 # [ 198.727567] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436747145worker2 # [ 198.739275] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)7146worker2 # [ 198.938755] nix-daemon[873]: accepted connection from pid 1785, user nix-grpc-daemon (trusted)7147worker2 # [ 198.950667] nix-daemon[873]: reaped child process 1788, status = succeeded7148worker2 # [ 199.119284] nix-daemon[873]: accepted connection from pid 1790, user nix-grpc-daemon (trusted)7149worker2 # [ 199.130494] nix-daemon[873]: reaped child process 1793, status = succeeded7150lb # [ 199.117502] niks3-server[1174]: 2026/09/21 13:00:48 INFO Received uploads request method=POST path=/api/pending_closures7151worker2 # [ 199.155565] nix-grpc-daemon[1562]: time=2026-09-21T13:00:48.218Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7152worker2 # [ 199.156928] nix-grpc-daemon[1562]: time=2026-09-21T13:00:48.218Z level=INFO msg="Uploading a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv (512B)"7153lb # [ 199.143206] niks3-server[1174]: 2026/09/21 13:00:48 INFO Registered completed upload object_key=nar/1qxv3rw2cwpq06kampqz63l69ci673kh4xz41xmflqybqycfk573.nar.zst7154lb # [ 199.153842] niks3-server[1174]: 2026/09/21 13:00:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign7155worker2 # [ 199.188226] nix-grpc-daemon[1562]: time=2026-09-21T13:00:48.251Z level=INFO msg="Uploading 1 narinfos"7156lb # [ 199.159247] niks3-server[1174]: 2026/09/21 13:00:48 INFO Signed narinfos id=70 count=17157lb # [ 199.165990] niks3-server[1174]: 2026/09/21 13:00:48 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.ls7158lb # [ 199.178107] niks3-server[1174]: 2026/09/21 13:00:48 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete7159lb # [ 199.182223] niks3-server[1174]: 2026/09/21 13:00:48 INFO Completed upload id=707160worker2 # [ 199.213853] nix-grpc-daemon[1562]: time=2026-09-21T13:00:48.277Z level=INFO msg="Upload complete. (461ms)"7161lb # [ 199.185909] niks3-server[1174]: 2026/09/21 13:00:48 INFO Registered completed upload object_key=a33ymyck2p14d34j7jl4zc8zvry6cysy.narinfo7162worker2 # [ 199.216149] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43674 duration_s=0 paths=1 nar_bytes_in=5127163worker2 # [ 199.221966] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436747164worker2 # [ 199.230873] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)7165worker2 # [ 199.233364] nix-daemon[873]: reaped child process 1783, status = succeeded7166worker2 # [ 199.238395] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)7167client # building '/nix/store/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv' on lb:50051...7168client # farm-a> worker2: building a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7169client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7170worker2 # [ 200.050840] nix-daemon[873]: reaped child process 1797, status = succeeded7171client # farm-a> LOG-farm-a-drain7172worker2 # [ 200.205627] nix-daemon[873]: accepted connection from pid 1813, user nix-grpc-daemon (trusted)7173worker2 # [ 200.217578] nix-daemon[873]: reaped child process 1816, status = succeeded7174lb # [ 200.203545] niks3-server[1174]: 2026/09/21 13:00:49 INFO Received uploads request method=POST path=/api/pending_closures7175worker2 # [ 200.241298] nix-grpc-daemon[1562]: time=2026-09-21T13:00:49.304Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7176worker2 # [ 200.242796] nix-grpc-daemon[1562]: time=2026-09-21T13:00:49.306Z level=INFO msg="Uploading rjdc0hf9mcchhia9m3zkqy1q586i3z3r-farm-a (128B)"7177lb # [ 200.241270] niks3-server[1174]: 2026/09/21 13:00:49 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7178lb # [ 200.253552] niks3-server[1174]: 2026/09/21 13:00:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign7179lb # [ 200.260399] niks3-server[1174]: 2026/09/21 13:00:49 INFO Registered completed upload object_key=log/a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv7180worker2 # [ 200.295868] nix-grpc-daemon[1562]: time=2026-09-21T13:00:49.359Z level=INFO msg="Uploading 1 narinfos"7181lb # [ 200.266300] niks3-server[1174]: 2026/09/21 13:00:49 INFO Signed narinfos id=71 count=17182lb # [ 200.271488] niks3-server[1174]: 2026/09/21 13:00:49 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.ls7183lb # [ 200.281060] niks3-server[1174]: 2026/09/21 13:00:49 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete7184lb # [ 200.287364] niks3-server[1174]: 2026/09/21 13:00:49 INFO Completed upload id=717185lb # [ 200.289125] niks3-server[1174]: 2026/09/21 13:00:49 INFO Registered completed upload object_key=rjdc0hf9mcchhia9m3zkqy1q586i3z3r.narinfo7186worker2 # [ 200.317211] nix-grpc-daemon[1562]: time=2026-09-21T13:00:49.380Z level=INFO msg="Upload complete. (272ms)"7187worker2 # [ 200.325912] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43674 duration_s=1 drv=a33ymyck2p14d34j7jl4zc8zvry6cysy-farm-a.drv assign_id=15 outputs=17188worker2 # [ 200.335872] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:49Z level=debug event=want client=7 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv cached=07189worker2 # [ 200.339151] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:49Z level=debug event=assigned drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv worker=192.168.1.4:50051 clients=17190worker2 # [ 200.347810] nix-daemon[873]: reaped child process 1795, status = succeeded7191worker2 # [ 200.379125] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436747192worker2 # [ 200.391682] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436747193worker2 # [ 200.404545] nix-grpc-daemon[1562]: time=2026-09-21T13:00:49.468Z level=INFO msg="All 1 paths already cached"7194client # copying 1 paths to 'lb:50051'...7195client # copying path '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' to 'lb:50051'...7196worker2 # [ 200.412594] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436747197worker2 # [ 200.420800] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)7198worker2 # [ 200.604444] nix-daemon[873]: accepted connection from pid 1822, user nix-grpc-daemon (trusted)7199worker2 # [ 200.617202] nix-daemon[873]: reaped child process 1825, status = succeeded7200worker2 # [ 200.786271] nix-daemon[873]: accepted connection from pid 1827, user nix-grpc-daemon (trusted)7201worker2 # [ 200.797683] nix-daemon[873]: reaped child process 1830, status = succeeded7202lb # [ 200.784400] niks3-server[1174]: 2026/09/21 13:00:49 INFO Received uploads request method=POST path=/api/pending_closures7203worker2 # [ 200.821476] nix-grpc-daemon[1562]: time=2026-09-21T13:00:49.885Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7204worker2 # [ 200.823338] nix-grpc-daemon[1562]: time=2026-09-21T13:00:49.886Z level=INFO msg="Uploading xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv (624B)"7205lb # [ 200.811160] niks3-server[1174]: 2026/09/21 13:00:49 INFO Registered completed upload object_key=nar/06skgfsii1iysx6lv8hy20iyys0q28pmz13rs51634x6bgrlvs7k.nar.zst7206lb # [ 200.821887] niks3-server[1174]: 2026/09/21 13:00:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign7207lb # [ 200.826766] niks3-server[1174]: 2026/09/21 13:00:49 INFO Signed narinfos id=72 count=17208worker2 # [ 200.859104] nix-grpc-daemon[1562]: time=2026-09-21T13:00:49.922Z level=INFO msg="Uploading 1 narinfos"7209lb # [ 200.833850] niks3-server[1174]: 2026/09/21 13:00:49 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.ls7210lb # [ 200.846496] niks3-server[1174]: 2026/09/21 13:00:49 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete7211worker2 # [ 200.881105] nix-grpc-daemon[1562]: time=2026-09-21T13:00:49.944Z level=INFO msg="Upload complete. (448ms)"7212lb # [ 200.851678] niks3-server[1174]: 2026/09/21 13:00:49 INFO Completed upload id=727213lb # [ 200.854898] niks3-server[1174]: 2026/09/21 13:00:49 INFO Registered completed upload object_key=xihm5w957gm8bh55zxlc6dnczbbl387q.narinfo7214worker2 # [ 200.883471] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43674 duration_s=0 paths=1 nar_bytes_in=6247215worker2 # [ 200.888713] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436747216worker2 # [ 200.897362] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)7217worker2 # [ 200.906136] nix-daemon[873]: reaped child process 1820, status = succeeded7218worker2 # [ 200.909806] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)7219client # building '/nix/store/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv' on lb:50051...7220client # farm-b> worker2: building xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7221client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7222client # farm-b> LOG-farm-b-drain7223worker2 # [ 202.736689] nix-daemon[873]: reaped child process 1834, status = succeeded7224worker2 # [ 202.920778] nix-daemon[873]: accepted connection from pid 1850, user nix-grpc-daemon (trusted)7225worker2 # [ 202.931218] nix-daemon[873]: reaped child process 1853, status = succeeded7226lb # [ 202.921457] niks3-server[1174]: 2026/09/21 13:00:51 INFO Received uploads request method=POST path=/api/pending_closures7227worker2 # [ 202.959806] nix-grpc-daemon[1562]: time=2026-09-21T13:00:52.023Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7228worker2 # [ 202.961326] nix-grpc-daemon[1562]: time=2026-09-21T13:00:52.024Z level=INFO msg="Uploading pmvxn7ymjj8pdn6gb65j5kqb45n38x0p-farm-b (176B)"7229lb # [ 202.956180] niks3-server[1174]: 2026/09/21 13:00:51 INFO Registered completed upload object_key=log/xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv7230lb # [ 202.970126] niks3-server[1174]: 2026/09/21 13:00:51 INFO Registered completed upload object_key=nar/17gyl0zgiivck81rhv539x50216ksc3nsf369ismwd4j1zwsdqrb.nar.zst7231lb # [ 202.980494] niks3-server[1174]: 2026/09/21 13:00:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign7232lb # [ 202.987463] niks3-server[1174]: 2026/09/21 13:00:51 INFO Signed narinfos id=73 count=17233worker2 # [ 203.019745] nix-grpc-daemon[1562]: time=2026-09-21T13:00:52.083Z level=INFO msg="Uploading 1 narinfos"7234lb # [ 202.990333] niks3-server[1174]: 2026/09/21 13:00:51 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.ls7235lb # [ 203.001149] niks3-server[1174]: 2026/09/21 13:00:51 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete7236lb # [ 203.007429] niks3-server[1174]: 2026/09/21 13:00:51 INFO Completed upload id=737237worker2 # [ 203.036952] nix-grpc-daemon[1562]: time=2026-09-21T13:00:52.100Z level=INFO msg="Upload complete. (318ms)"7238lb # [ 203.009175] niks3-server[1174]: 2026/09/21 13:00:51 INFO Registered completed upload object_key=pmvxn7ymjj8pdn6gb65j5kqb45n38x0p.narinfo7239worker2 # [ 203.045328] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43674 duration_s=2 drv=xihm5w957gm8bh55zxlc6dnczbbl387q-farm-b.drv assign_id=16 outputs=17240worker2 # [ 203.056156] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:52Z level=debug event=want client=7 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv cached=07241worker2 # [ 203.059224] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:52Z level=debug event=assigned drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv worker=192.168.1.4:50051 clients=17242worker2 # [ 203.067336] nix-daemon[873]: reaped child process 1832, status = succeeded7243worker2 # [ 203.098337] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436747244worker2 # [ 203.110622] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436747245worker2 # [ 203.123395] nix-grpc-daemon[1562]: time=2026-09-21T13:00:52.186Z level=INFO msg="All 2 paths already cached"7246client # copying 1 paths to 'lb:50051'...7247client # copying path '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' to 'lb:50051'...7248worker2 # [ 203.131601] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436747249worker2 # [ 203.142133] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)7250worker2 # [ 203.324768] nix-daemon[873]: accepted connection from pid 1859, user nix-grpc-daemon (trusted)7251worker2 # [ 203.336656] nix-daemon[873]: reaped child process 1862, status = succeeded7252worker2 # [ 203.516300] nix-daemon[873]: accepted connection from pid 1864, user nix-grpc-daemon (trusted)7253worker2 # [ 203.527616] nix-daemon[873]: reaped child process 1867, status = succeeded7254lb # [ 203.516972] niks3-server[1174]: 2026/09/21 13:00:52 INFO Received uploads request method=POST path=/api/pending_closures7255worker2 # [ 203.554522] nix-grpc-daemon[1562]: time=2026-09-21T13:00:52.618Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7256worker2 # [ 203.556436] nix-grpc-daemon[1562]: time=2026-09-21T13:00:52.619Z level=INFO msg="Uploading q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv (752B)"7257lb # [ 203.550913] niks3-server[1174]: 2026/09/21 13:00:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign7258lb # [ 203.554538] niks3-server[1174]: 2026/09/21 13:00:52 INFO Registered completed upload object_key=nar/1m4vpncprrwddjpk4qry79vzd9g39s9pjs6874i2aarpm7ax0ykq.nar.zst7259lb # [ 203.560062] niks3-server[1174]: 2026/09/21 13:00:52 INFO Signed narinfos id=74 count=17260worker2 # [ 203.592586] nix-grpc-daemon[1562]: time=2026-09-21T13:00:52.655Z level=INFO msg="Uploading 1 narinfos"7261lb # [ 203.567384] niks3-server[1174]: 2026/09/21 13:00:52 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.ls7262lb # [ 203.578258] niks3-server[1174]: 2026/09/21 13:00:52 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete7263lb # [ 203.584703] niks3-server[1174]: 2026/09/21 13:00:52 INFO Completed upload id=747264worker2 # [ 203.614255] nix-grpc-daemon[1562]: time=2026-09-21T13:00:52.677Z level=INFO msg="Upload complete. (460ms)"7265lb # [ 203.587050] niks3-server[1174]: 2026/09/21 13:00:52 INFO Registered completed upload object_key=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s.narinfo7266worker2 # [ 203.617755] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43674 duration_s=0 paths=1 nar_bytes_in=7527267worker2 # [ 203.622778] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436747268worker2 # [ 203.632076] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)7269worker2 # [ 203.636672] nix-daemon[873]: reaped child process 1857, status = succeeded7270worker2 # [ 203.641840] nix-daemon[873]: accepted connection from pid 1488, user nix-grpc-daemon (trusted)7271client # building '/nix/store/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv' on lb:50051...7272client # farm-top> worker2: building q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7273client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7274client # farm-top> LOG-farm-top-drain7275worker2 # [ 204.866735] nix-daemon[873]: reaped child process 1871, status = succeeded7276worker2 # [ 205.041566] nix-daemon[873]: accepted connection from pid 1887, user nix-grpc-daemon (trusted)7277worker2 # [ 205.053108] nix-daemon[873]: reaped child process 1890, status = succeeded7278lb # [ 205.042698] niks3-server[1174]: 2026/09/21 13:00:53 INFO Received uploads request method=POST path=/api/pending_closures7279worker2 # [ 205.081216] nix-grpc-daemon[1562]: time=2026-09-21T13:00:54.143Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7280worker2 # [ 205.082570] nix-grpc-daemon[1562]: time=2026-09-21T13:00:54.143Z level=INFO msg="Uploading hgmfp1yw6m31jn6cpma19ikl6dy1b17r-farm-top (232B)"7281lb # [ 205.078554] niks3-server[1174]: 2026/09/21 13:00:53 INFO Registered completed upload object_key=nar/1xin0j0xlljjqnd952hnxy8yw45d9j9qpj7m241vv6cckfq8kk4p.nar.zst7282lb # [ 205.084115] niks3-server[1174]: 2026/09/21 13:00:53 INFO Registered completed upload object_key=log/q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv7283lb # [ 205.093681] niks3-server[1174]: 2026/09/21 13:00:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign7284lb # [ 205.098992] niks3-server[1174]: 2026/09/21 13:00:54 INFO Signed narinfos id=75 count=17285worker2 # [ 205.131305] nix-grpc-daemon[1562]: time=2026-09-21T13:00:54.194Z level=INFO msg="Uploading 1 narinfos"7286lb # [ 205.104335] niks3-server[1174]: 2026/09/21 13:00:54 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.ls7287lb # [ 205.113532] niks3-server[1174]: 2026/09/21 13:00:54 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete7288lb # [ 205.118858] niks3-server[1174]: 2026/09/21 13:00:54 INFO Completed upload id=757289worker2 # [ 205.148392] nix-grpc-daemon[1562]: time=2026-09-21T13:00:54.211Z level=INFO msg="Upload complete. (298ms)"7290lb # [ 205.121800] niks3-server[1174]: 2026/09/21 13:00:54 INFO Registered completed upload object_key=hgmfp1yw6m31jn6cpma19ikl6dy1b17r.narinfo7291worker2 # [ 205.154587] nix-grpc-daemon[1488]: ts=2026-09-21T13:00:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43674 duration_s=1 drv=q6a73jmqx0c0v5nlki9f1fnmcls5gx5s-farm-top.drv assign_id=17 outputs=17292worker2 # [ 205.167989] nix-daemon[873]: reaped child process 1869, status = succeeded7293client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.pem&client-key=/nix/store/gg5dnjqvsbzhmrc2lb82r8kx0z2r1xcp-farm-certs/ci.key' --eval-store auto -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain >&2, in 6.81 seconds)7294client: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath7295client: (finished: must succeed: nix eval --raw -f /nix/store/kh0qcgyv9dmrj3hks77rdgj728ar81vy-job.nix --argstr tag drain outPath, in 0.23 seconds)7296worker1: must succeed: rm /nix/.rw-store/fill7297worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.13 seconds)7298worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7299worker1 # [ 206.345534] nix-grpc-daemon[1721]: ts=2026-09-21T13:00:55Z level=info event=healthy reason=min_free7300worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.11 seconds)7301(finished: subtest: low disk drains a worker and builds go to the other, in 10.40 seconds)7302(finished: run the VM test script, in 207.63 seconds)7303test script finished in 207.87s7304cleanup7305kill QemuMachine (pid 49)7306client # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7307kill QemuMachine (pid 50)7308lb # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7309kill QemuMachine (pid 52)7310worker1 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7311kill QemuMachine (pid 51)7312worker2 # qemu-system-x86_64: terminating on signal 15 from pid 39 (/nix/store/b5bpi6zfajzzrwwpgba2q6li3nnya4bs-python3-3.14.7/bin/python3.14)7313(finished: cleanup, in 1.02 seconds)7314additionally exposed symbols:7315 client, lb, worker1, worker2,7316 vlan1,7317 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh